Part V - Improve and Refresh Research Methods and Practice

RMP.9 — Decide Whether and How to Improve Research Methods and Practice

Type: Research Method Practice practitioner method pattern

RMP.9:0 - Use This When

Use this pattern when actual research work, criticism, a changed instrument or analytic tool, a new receiving situation, or learning within a field raises a concrete question about retaining or improving a research Method and its practice. Begin with the research use and what has become difficult or newly possible. Identify the object that needs attention and ask whether the available basis already supports retaining the Method or making a bounded correction.

The practical gain is an adequate Method kept in use, a justified change that reaches its actual users, or a clear limitation on an unsupported use. A known calculation error can be corrected without starting a Method-efficacy study. A materially changed research situation can justify a comparison when its possible results can change the decision and the whole effort is worthwhile.

Research-Method improvement is the branch of wider Method improvement governed here. Its particular concerns are the knowledge question, research validity and criticism, field-specific interpretation, actual research material and the claims that depend on the Method. Method Engineering supplies reusable identity, trial, fit, worth, variant and introduction contributions where needed.

A disappointing finding, an unfamiliar technique, reviewer disagreement or a software release is a cue to inspect the question, not proof that the Method has failed. If only a particular analysis or knowledge claim needs correction, use RMP.5 or RMP.8. Return routine equipment recovery, institutional administration and permissions to their actual owners. Investigating a Method enters new research only when the RMP.1 research-membership conditions pass.

RMP.9:0.1 - Working Terms

RMP.9:1 - Problem Frame

A research team can encounter the same visible failure for different reasons. An uncertainty estimate can be wrong because a formula was implemented incorrectly, because observations violate its assumptions, or because the intended quantity changed. Correcting software addresses the first problem. New data do not automatically repair the second, and ranking estimators before settling the target can obscure the third.

Professional traditions also differ in what improvement means. A statistical comparison can examine coverage under relevant generating conditions. Reflexive qualitative analysis can need a more coherent interpretation, while a historical argument can need correction of a source transcription. A criterion useful for one purpose can distort another.

Improvement therefore starts with a situated question and a real alternative. Existing adequate practice, a limited correction, another Method, a situation-dependent variant and stopping an unsupported use can each be the strongest supported answer. Further inquiry is one possible contribution to that decision.

RMP.9:2 - Problem

When every criticism is treated as a Method defect, teams replace useful practice and commission unnecessary trials. When every weakness is attributed to execution, an unsuitable Method survives. When improvement is judged by speed, agreement, novelty or a completed checklist alone, the research contribution can deteriorate while its chosen indicator improves.

There is a further trap in evidence acquisition. An unresolved comparison can generate a new protocol and reporting burden before anyone asks what its result could change or whether the comparison is attainable. Conversely, declining an expensive study can be mistaken for evidence that the present Method is adequate. The decision needs both a useful completion and an honest boundary on unsupported use.

RMP.9:3 - Forces

RMP.9:4 - Solution

RMP.9:4.1 - Locate the question and try the supported present answer

Name the research use, the difficulty or opportunity, and what materially different answer would change in practice. Recover the relevant current Method, its description, actual application and receiving conditions far enough to distinguish the object at issue.

Check whether the observed failure comes from the reusable Method, its interpretation or implementation, a local deviation, unavailable capability or support, or an affected analysis or claim. Use the available evidence to make that distinction. A symptom that remains unexplained leaves the diagnosis open; it does not license a preferred replacement.

When an adequate present basis settles the question, use it. Retain the Method for its supported conditions, make the determinate correction, or return the bounded result to its owner. Verify the correction at the affected object and preserve the limits needed by whoever uses the result. Communicate this conclusion in the ordinary work result when that is sufficient.

First useful example. If a suitable Method calls for sample variance with denominator n−1 but its implementation uses n, fix and check that calculation. The Method can remain unchanged. Reanalyse the dependent results that used the wrong value; no new participants or efficacy trial are needed to determine the specified correction. The full case below shows the calculation and its reach.

RMP.9:4.2 - Compare alternatives for the same research use

When the decision remains open, identify attainable alternatives. These can include retaining adequate practice, repairing its description or support, changing the Method, keeping different variants for distinguishable situations, or stopping a use that lacks a defensible basis. Include an alternative because it can answer the actual difficulty, not to fill a quota.

Preserve the intended quantity, interpretation or other research purpose across the comparison. Make explicit any change of question that makes alternatives appear to disagree. Assess their capacity to withstand the criticism relevant to that purpose, their fit to the actual material and receiving conditions, and their consequences for users of the research.

Compare material burdens and who bears them. Depending on the situation these include access, participant time or exposure, equipment, software, competence, design, computation, reading, interpretation, documentation, transition, maintenance, recovery and displaced research. Include the effort of proposing and explaining the comparison itself. Record consequential differences at the detail needed for this decision.

A hard validity, ethical or access limit cannot be offset by an attractive average score. Scientific value can justify substantial or long-horizon work; immediate operational return is not the sole measure of worth. Keep real trade-offs and incomparable positions visible when no warranted single ranking exists.

RMP.9:4.3 - Decide whether further evidence is worth obtaining

Before designing a new trial or study, ask:

• What materially different result could change the present decision?
• Could a discriminating comparison be obtained under the actual access, capability, material and time constraints?
• Why is its whole burden warranted by that possible change?

These questions select acquisition; they do not require a new form. If existing adequate evidence supports retention or a bounded correction, finish with that answer. If a consequential decision remains open and an informative comparison is attainable and worthwhile, select the needed work.

Otherwise preserve the strongest supported position and its limits. Restrict or stop the unsupported use where the missing basis prevents it. “Further evidence is not worth obtaining now” and “existing evidence is sufficient” answer different questions. Neither an unselected experiment nor an unchanged adequate Method needs an omission certificate.

For a selected comparison, choose the research or Method Engineering contribution that can obtain the required evidence. Preserve the difference among the proposed change, intended trial, actual work, observations, inference and Method decision. A trial plan is not an obtained improvement.

In a statistical simulation, define the aim, generating situations, common estimand, candidate procedures and performance measures before judging results. Account for Monte Carlo uncertainty and failed or non-convergent outputs. Performance calculated only among successful runs can conceal a consequential weakness. Use the full observed result at its applicable reach; a favourable selected case does not establish general fit.

For a qualitative, historical or mixed inquiry, use the criticism and evidence appropriate to that purpose. Do not introduce a statistical benchmark merely to make unlike research traditions share one indicator. A reporting practice can improve inspectability while leaving the Method’s substantive adequacy unresolved.

RMP.9:4.4 - Retain or introduce the supported change

State what practitioners retain or change and the conditions under which the decision applies. Distinguish a semantic Method revision from a description correction, software repair or local deviation. A semantic change raises a variant question; previous qualification evidence applies only where its assumptions and reach still match.

For a selected change, make its consequences usable in actual practice. Supply the description, competence, tools or other support that is actually missing. Introduce it within the supported conditions, retain consequential provenance and recovery options, and distinguish its announcement, actual use and subsequent observed effects.

A team already capable of making a small correction does not automatically need a new training programme. A change with wider consequences may need substantial preparation and observation. Select later observation when it can expose a material failure or revise a live decision; do not turn every local repair into indefinite monitoring.

A Method retained for one situation may remain unsuitable for another. A justified branch names the situation discriminator and the evidence each variant can use. Do not turn an undocumented adaptation into a qualified variant simply by adopting a new name.

RMP.9:4.5 - Return the actual consequence and finish

Identify the result that depends on the retained or changed Method. Reopen only that dependency and preserve unaffected work. A revised measurement relation may change operationalization and analysis; a corrected extraction may change the trace and historical claim; a changed synthesis procedure may change a corpus contribution. The direct relations below identify the RMP destinations.

ResearchMethodPracticeRevision communicates the research use, actual subject, supported disposition, necessary basis and limits, change in practice and affected receiving result. It may be a concise conclusion in existing research or Method work. A decision to obtain further evidence identifies that selected work and the still-unresolved qualification; it does not claim an improvement already obtained.

If the Method question becomes a new research question, return to RMP.1’s four conditions: unresolvedness against a qualified current frame, non-routine choice under genuine uncertainty, a systematic criticism-ready trace and a bounded communicable episteme contribution. Routine verification remains with its actual owner when those conditions do not hold.

Recognition and assurance remain distinct. A failure, dispute or changed tool helps recognize the question. Adequate retention, a verified correction, a warranted transfer and a causal claim of improvement need the evidence appropriate to their different reaches. Applying this pattern or completing a report supplies none of those conclusions by itself.

RMP.9:5 - Archetypal Grounding

The cases below are constructed teaching cases. Their stated premises define the situations in which the choices follow.

RMP.9:5.1 - Correct the calculation and retain the Method

A team’s accepted Method estimates sample variance for independent observations by dividing the squared-deviation sum by n−1. The receiving conditions still fit this Method. A software default instead divides by n.

For observations 2, 4, 6 and 8, the mean is 5 and the squared-deviation sum is 9 + 1 + 1 + 9 = 20. The specified sample variance is 20/3; the implementation reports 5. The difference is a determinate implementation defect under the case premises.

Correct the denominator and verify the calculation and its dependent uncertainty output. The result is: retain the Method for the named conditions, repair its implementation, identify analyses that used the wrong value, and return consequential corrections through RMP.5 and RMP.8. Unaffected analyses remain valid at their existing reach.

Additional participants or a new efficacy trial would not determine this already specified correction. The known-answer check establishes the corrected calculation; it does not qualify another estimator or a changed receiving situation. This is a completed useful ResearchMethodPracticeRevision.

RMP.9:5.2 - Select a worthwhile comparison without inventing its result

A programme now estimates an equally weighted mean across independent sites, with repeated observations within each site. Site sample sizes differ and observations within a site are dependent. The old pooled-observation analysis does not answer the intended site-level question with defensible uncertainty under the case premises.

The team first fixes the target. Two attainable candidates use the same mean of site means but construct uncertainty differently: a t interval from independent site means and an interval obtained by resampling whole sites. The names of these procedures do not settle their adequacy when there are few sites, skewed outcomes and plausible heterogeneity.

A qualified statistician, existing simulation code and a bounded two-day allocation make a discriminating comparison attainable. The programme will reuse the chosen analysis across several studies, and a wrong uncertainty claim would materially affect that work. Include design, code checks, computation, failed runs, Monte Carlo uncertainty, interpretation and transition in the burden; the two-day availability is a case premise, not a general cost estimate.

Select the simulation around relevant generating situations, the shared estimand and the performance required by the intended use. Preserve adverse and inconclusive results. Both candidates may prove inadequate for the proposed reach, leading to a narrower claim or stopped use.

The completed result here is a justified decision to obtain evidence. Method qualification and the choice of a winning procedure remain unresolved until actual results exist. If the access, capability or worth premises change, reconsider this acquisition decision.

RMP.9:5.3 - Repair interpretation within reflexive thematic analysis

A reflexive thematic analysis team receives a request to improve quality by maximizing agreement among coders. In this constructed case, the actual weakness is a topic list that does not develop a coherent interpretation. Disagreement alone does not identify a failure of this approach.

Rework the analysis through coherent interpretation and reflexive discussion, and judge the resulting themes against that purpose. A compulsory reliability study would address another question. If the research purpose instead requires repeatable fixed-category counts, select and qualify the corresponding Method and criteria. This distinction applies to reflexive thematic analysis; agreement criteria can be appropriate in other Methods.

RMP.9:5.4 - Correct source extraction without certifying a new tool

An OCR tool repeatedly omits marginal dates in a bounded historical collection. Source images are available. Recover the missing material, correct the affected transcription and chronology, and preserve the relation between the source and the resulting interpretation. This repairs the extraction and its consequences without automatically replacing the historical Method.

A replacement tool intended for a materially different handwriting collection raises another question. Select a comparison only when it can change a consequential source use and access, expertise and effort make it worthwhile. Faster extraction alone does not establish fidelity. If originals are unavailable, preserve the resulting limitation on the argument; declining a new experiment does not make the transcription reliable.

RMP.9:5.5 - Preserve the distinct contributions in mixed research

A project combines a fixed-category count with an interpretive account. A coding revision changes both what is counted and what the categories mean. Reassess the affected count, interpretation and integration relation at their own purposes. A gain in counting consistency cannot by itself settle the interpretation, while an interpretive insight cannot validate an altered measurement.

Return the affected operational, analytic or synthesis contribution through RMP.3, RMP.5 or RMP.7. Retain the unaffected material and make any changed claim visible to its actual recipient. The two contributions need a defensible relation, not an invented common score.

RMP.9:6 - Bias-Annotation

Statistical examples make numerical errors and comparison criteria easy to inspect. That convenience can obscure interpretive, historical and community research whose criticism takes another form. The unlike cases preserve those differences, but do not stand in for every field’s professional knowledge.

Method improvement can favour novelty, visible activity or readily counted gains. Retention and small corrections can receive less attention, while a cheap-exit preference can hide a genuine need for evidence. The relevant counterweight is the named research use, current alternatives, full burden and the limit on unsupported application.

RMP.9:7 - Conformance Checklist

Check the conclusion at its claimed reach, using the ordinary work result:

• Does the research use and observed difficulty identify the actual subject of retention or change?
• Does an existing adequate basis already settle the question or support a determinate correction?
• If further evidence is selected, can it change the decision, is it attainable, and does its whole burden make sense?
• If evidence remains insufficient, is the unsupported use restricted rather than silently qualified?
• Does the conclusion distinguish proposed change, actual performance, qualification and observed consequence?
• Does a changed Method retain the necessary variant, provenance, receiving-condition and recovery distinctions?
• Can the actual consumer apply the warranted consequence while unaffected results and receiving authority remain intact?

RMP.9:8 - Common Anti-Patterns and How to Avoid Them

RMP.9:9 - Consequences

A team can finish with useful retention or a small verified correction, while consequential Method weaknesses still receive investigation when its value and feasibility warrant it. Users of a revised Method receive its supported conditions and the specific consequences for their research.

The price is careful diagnosis and judgement. A large Method change may require substantial evidence, capability, transition and maintenance work. The pattern identifies that work; it does not make it inexpensive or supply professional competence.

Reopen when criticism, instruments, tools, receiving conditions, relevant sources, practical burdens or observed use change the supported decision. A later discovery affecting one analysis need not invalidate every use of the Method.

RMP.9:10 - Rationale

Research-Method improvement needs a research-specific connection between a Method difficulty, its criticism and the knowledge contributions that depend on it. General Method Engineering already provides reusable distinctions for trial, fit, worth, variants and introduction. Using those contributions at the required scope avoids reconstructing a full Method lifecycle for every correction.

The order matters. Identifying the actual object and trying the supported present answer prevent unnecessary acquisition before its costs arise. Separating evidence sufficiency from acquisition worth then protects the opposite boundary: a justified decision to stop looking does not establish an unsupported claim.

RMP.9:11 - SoTA-Echoing

The working question is how to improve research practice when criticism leaves a Method decision open, without spending effort on evidence that cannot change the answer. The best-known answer selected here combines situated comparison with a prior decision about whether further evidence is worth obtaining. Method Engineering supplies reusable identity, trial, fit, worth, variant and introduction distinctions. ME.14:4.1–4.2 compares feasible courses and their full burdens, permits an adequate present choice or a retained set with its limitation, and selects further inquiry only when its obtainable result could change the choice enough to warrant the whole burden. RMP.9 adopts these stopping and acquisition distinctions and adapts the comparison to the research question, criticism requirement and affected research result in §4.3. A sufficient existing research account can carry that decision without a separate ME.14 record. At the effort needed to diagnose §5.1’s specified calculation error, retention plus verified repair already answers the use; a further efficacy study adds no discrimination. Where qualification is unresolved, preserve the uncertainty and restrict unsupported use. Reopen this source use if Method Engineering changes the relied-on stopping or acquisition conditions; reconsider a research comparison when its obtainable result becomes worth the whole burden.

For a simulation already selected, the question is which procedure answers the intended research target under the relevant conditions. RMP.9 adopts the common-target design in Morris, White and Crowther (2019), refined by Williams et al. (2024) on inspectable planning/reporting and Pawel et al. on missing outputs and non-convergence. These sources support the selected line over a comparison that silently ranks only successful runs: different failure rates can make that ranking answer another question. In §§4.3 and 5.2, preserve the common estimand, generating situations, performance criteria, Monte Carlo uncertainty and failed runs. This deliberately accepts the effort of diagnosing and interpreting failures within the comparison’s whole burden; apparent simplicity is insufficient reason to discard them. Their treatment follows the study’s aim. The sources discipline an obtained comparison, not its necessity or a universally winning Method. Reopen when a changed target, newly observed failure or stronger treatment of missing outputs changes the applicable comparison.

For reflexive thematic analysis, the question is how to repair weak interpretation while preserving that research purpose. RMP.9 adopts methodological coherence from Braun and Clarke’s RTARG contribution (2024) and their quality guidance. Section 5.3 rejects compulsory coder agreement as the repair for an undeveloped topic list: agreement can leave the interpretive defect intact. The selected work spends analytic effort on developing a coherent interpretation and reflexive scrutiny. This choice forgoes an agreement indicator that serves a different purpose; fixed-category counting would reopen the Method and criteria. Reporting guidance alone does not establish interpretive quality.

For historical extraction, RMP.9 adapts the AHA professional statement’s source-integrity and recoverable-trail requirements to §5.4’s missing marginal dates. Comparing extraction tools by speed alone cannot establish fidelity to the available originals. Correcting the bounded extraction and chronology preserves that criterion and its receiving argument, while accepting the effort of checking sources. This is a reasoned application of a professional constraint, not evidence that one OCR tool is superior. A different collection, unavailable originals or a newly attainable tool comparison reopens the applicable correction or limitation.

These comparisons preserve different research purposes rather than ranking statistical, reflexive and historical work on one scale. Their synthesis and constructed cases explain the selected action and trade-offs; they are not measured evidence that deploying RMP.9 improves research outcomes.

RMP.9:12 - Relations

RMP.9:End