Jump to text
On this page
Copy and source
Published source confirmed at last check

Source changed 2026-11-16 21:18:08 UTC · snapshot created 2026-11-16 21:19:40 UTC · last check 2026-11-17 19:15:10 UTC

Part V - Improve and Refresh Research Methods and Practice

RMP.9 — Decide Whether and How to Improve Research Methods and Practice

Status: Stable

Type: Research Method Practice practitioner method pattern

RMP.9:0 - Use This When

Use this method when research work, criticism, a change in tools or a new use raises a question about the suitability of a research Method or the way researchers practise it. State the use and the difficulty or opportunity. Determine what the available evidence says about the Method, its description, its implementation or the conditions in which it was used.

The result can explain what happened, support retaining a Method, identify a correction or show where a use lacks support. If the task includes introducing a change, establish what its users need. A specified calculation error can be corrected directly. A new comparison is worth considering when its attainable results matter enough to justify the full effort.

Research-Method improvement connects a knowledge question, criteria of validity and criticism, field-specific interpretation, research material and the claims that depend on the Method. FPF Core supplies methods for identifying, trying and comparing Methods, assessing fit and worth, and distinguishing variants. Use A.3.1.MI when introducing a Method into practice.

A disappointing finding, an unfamiliar technique, reviewer disagreement or a software release can prompt an inquiry. Establish which part of the work is affected. Use RMP.5 to examine a particular analysis or RMP.8 to revise a knowledge claim. Use the relevant operations or institutional practice for equipment recovery, administration and permissions. Apply the four RMP.1 conditions when asking whether the inquiry into a Method qualifies as new research.

RMP.9:0.1 - Working Terms

TermMeaning for this use
research MethodA repeatable way of investigating, analysing, interpreting or criticizing a research question. Its suitability depends on the question and conditions of use.
Method descriptionInstructions or an account of how to use a Method. A correction can clarify the account while preserving what practitioners are to do.
implementation or supportSoftware, an instrument, procedural support or another means of applying the Method. A defect in it can explain the failure of an otherwise suitable Method.
semantic Method changeA change in what practitioners are to do that can alter the Method’s applicability or the claims it supports. Determine whether this introduces a variant and which qualifications carry over.
supported retentionKeeping a Method or practice for conditions in which adequate evidence supports its use. Examine its fit separately when those conditions change.
ResearchMethodPracticeRevisionAn account of the supported outcome. It can identify a retained Method, a correction, a changed Method, a restriction or a discontinued use.

RMP.9:1 - Problem Frame

The same error in an uncertainty estimate can have different explanations. The implementation may calculate the wrong formula, the observations may violate its assumptions, or the quantity of interest may have changed. Check a faulty implementation against the specified formula, examine the assumptions against the observations, and settle the target before ranking estimators for it.

Professional traditions also differ in what improvement means. A statistical comparison can examine coverage under relevant generating conditions. Reflexive qualitative analysis can need a more coherent interpretation, while a historical argument can need correction of a source transcription. A criterion useful for one purpose can distort another.

Compare the alternatives that bear on the question. Available evidence may support keeping the present practice, making a limited correction, adopting a different Method, using variants in distinguishable situations or stopping an unsupported use. Further inquiry can supply evidence when the choice is open.

RMP.9:2 - Problem

Teams can replace a useful Method after a criticism of its implementation, or excuse a genuine Method defect as poor execution. Speed, agreement or novelty can improve while the research result deteriorates. Check the indicator against the research purpose before using it to judge improvement.

An unresolved comparison can cost time for design, execution and reporting. Estimate what its results could establish and whether the comparison is attainable. If the cost rules it out, judge the present Method from the evidence available. An untested use can remain unsupported.

RMP.9:3 - Forces

ForceTension
Preserve useful practiceA familiar Method may merit retention. Judge its fit from evidence about the use in question.
Locate the defectA similar symptom can arise from the Method, its description or implementation, the practitioner’s capability or the claim made from the result. Each calls for a different repair.
Criticism within the research purposeShared demands for inspectability coexist with different statistical, interpretive and historical criteria.
Evidence and whole burdenA discriminating comparison can improve many later studies, while its design, conduct and uptake consume real research capacity.
Change and continuityUsers introducing a revision need access to the change, its provenance and the conditions for reusing earlier qualifications.

RMP.9:4 - Solution

RMP.9:4.1 - Locate the question and try the supported present answer

Identify the research use, the question about the Method and the evidence that could affect the answer. Establish the relevant Method, its description, how it was or would be applied, and the receiving conditions as needed to distinguish the object at issue. For a proposed change, identify what its introduction would alter in practice.

Use available evidence to distinguish the reusable Method from its interpretation or implementation, a local deviation, a gap in capability or support, and an affected analysis or claim. If several explanations fit an observed failure, say which remain possible and what each implies.

Use an adequate existing result within its conditions. It may settle a question about a past application, warrant retention or specify a correction. If a correction is commissioned, check it at the affected object and give the users of its result the conditions on which it depends.

First useful example. Suppose a suitable Method specifies the sample variance with denominator n−1 and the implementation uses n. If the error is still present, correct it and check the calculation against the formula. A saved check of the repaired calculation may already support an account of the earlier correction. When correcting dependent work, identify the results that relied on the wrong value and reanalyse those whose conclusion or basis is affected. Section 5.1 works through the numbers.

RMP.9:4.2 - Compare alternatives for the same research use

For an open decision, compare the attainable courses that address the difficulty. They may include retaining the practice, repairing its description or support, changing the Method, keeping variants for distinguishable situations or stopping a use that lacks a defensible basis. A supported conclusion on one course can answer a question about that course.

Compare alternatives for the same quantity, interpretation or research purpose. If they answer different questions, state the difference. Examine their resistance to the relevant criticism, their fit to the material and receiving conditions, and the implications for users.

Compare material burdens and identify who bears them. Include relevant demands on access, participant time or exposure, equipment, competence, software, design, computation, reading, interpretation, documentation, transition, maintenance and recovery. Account for displaced research and for the work of proposing and explaining the comparison.

Exclude alternatives that violate binding validity, ethical or access conditions. Count long-term scientific value where it matters. Where the accepted criteria leave alternatives incomparable, explain the trade-offs.

RMP.9:4.3 - Decide whether further evidence is worth obtaining

Before designing a new trial or study, ask:

  • What could the new evidence establish or change about the claim, its support or the decision?
  • Could a discriminating comparison be obtained under the actual access, capability, material and time constraints?
  • Would that contribution justify the full effort of obtaining and using the evidence?

Use sufficient existing evidence for retention, a specific correction or a conclusion about an earlier use. If an important question remains, select a comparison that can answer it within the available conditions and whose contribution warrants the work.

If a further comparison is out of reach or costs more than its result warrants, state what the relevant evidence supports and where its support ends. That can finish an account of an earlier use. If advice or a restriction is called for, identify the unsupported use and act within the authority for that work. Declining a new study leaves its possible result open.

For a selected comparison, choose a research or Core Method-development method that can obtain the needed evidence. State which existing or proposed methods or uses are being compared and what evidence could distinguish them in the selected comparison. For a completed trial, establish what was done and observed, the inference those observations support and any resulting Method decision.

For a simulation, define its aim, generating situations, common estimand, candidate procedures and performance measures. Account for Monte Carlo uncertainty and for missing or non-convergent outputs. An analysis restricted to successful runs can favour a procedure that often fails. Report what happened across the runs and limit the performance claim to the conditions examined.

Use the standards of criticism appropriate to a qualitative, historical or mixed inquiry. For example, improved reporting may make an interpretation inspectable. Judge the interpretation itself against the research question and its evidence.

RMP.9:4.4 - Retain or introduce the supported change

State what practitioners retain or change and the conditions that support that conclusion. Distinguish a revision of what the Method requires from a correction to its description or software and from a local deviation. For a changed Method, assess whether it forms a variant and which earlier evidence still applies.

When introducing a selected change, supply the missing instructions, capability, tools or other support. Introduce it under the supported conditions and keep the provenance and means of recovery needed by users. Distinguish notification, actual use and observed effects.

A team with the necessary capability can make a small correction directly. A wider change may call for preparation or follow-up. Select a later observation when its attainable result could reveal an important failure, clarify an effect or affect a decision.

When two variants serve different situations, give the feature that distinguishes those situations and the evidence supporting each use. To qualify a local adaptation, establish what changed and where the evidence applies.

RMP.9:4.5 - Return the actual consequence and finish

Identify any results that relied on the affected part of the Method. If the change alters their basis or warranted scope, reassess that part of each result and retain the work that still holds. A revised measurement relation can affect operationalization and analysis, a corrected extraction can affect a historical claim, and a changed synthesis procedure can affect a corpus finding. Section 12 gives the corresponding methods.

In a ResearchMethodPracticeRevision, give the research use, the object examined, the supported outcome and its basis and limits. Include the change to practice and consequences for receiving work when present. If the decision is to obtain further evidence, identify the selected work and the open qualification question it addresses.

If investigating the Method becomes a research question, test the whole inquiry against the four conditions in RMP.1:4.2. A fully specified protocol can investigate a disputed knowledge claim. A check of a known correction against an accepted rule can finish as ordinary Method work.

A failure, a dispute or a changed tool can raise the question. Use evidence of fit for retention, a check against the applicable criterion for a correction, evidence of applicability for transfer, and grounds for attributing an improvement to the change when making a causal claim.

RMP.9:5 - Archetypal Grounding

The following cases use stated assumptions to show when different answers are warranted.

RMP.9:5.1 - Correct the calculation and retain the Method

A team’s accepted Method estimates sample variance for independent observations by dividing the squared-deviation sum by n−1. The receiving conditions still fit this Method. A software default instead divides by n.

For observations 2, 4, 6 and 8, the mean is 5 and the squared-deviation sum is 9 + 1 + 1 + 9 = 20. The specified sample variance is 20/3; the implementation reports 5. The difference is a determinate implementation defect under the case premises.

Correct the denominator and check both the sample variance and any dependent uncertainty calculation. For these observations, the correction changes 5 to 20/3. Retain the Method for the stated conditions. Trace the analyses that used the wrong denominator on any dataset and return consequential corrections through RMP.5 and RMP.8.

The known-answer check can establish that the implementation now produces the specified value for these observations. Qualifying another estimator or using this Method under different conditions requires evidence for that question.

RMP.9:5.2 - Select a worthwhile comparison without inventing its result

Suppose a programme now needs an equally weighted mean across independent sites. Site sample sizes differ, and observations within a site are dependent. Pooling observations weights sites by sample size. Under these conditions the former analysis does not provide the required weighting or uncertainty calculation.

Both candidate procedures use the mean of the site means. One constructs a t interval from the independent site means. The other resamples whole sites to construct an interval. For few sites, skewed outcomes and plausible heterogeneity, compare how the intervals perform under conditions relevant to their intended use.

Suppose a qualified statistician and existing code put the comparison within a two-day allocation, including design, code checks, computation, investigation of failed runs, Monte Carlo uncertainty and interpretation. Include the cost of introducing a chosen procedure in the decision. The programme would reuse the analysis across several studies whose uncertainty claims matter.

Select a simulation that compares performance for the shared estimand across the relevant generating situations. Include adverse and inconclusive results. If neither procedure meets the requirement, consider a narrower supported claim or stop the proposed use.

These premises justify obtaining the comparison. Decide which procedures fit the proposed use when adequate performance evidence is available. Reconsider the investigation if its access, capability, cost or expected contribution changes.

RMP.9:5.3 - Repair interpretation within reflexive thematic analysis

A reflexive thematic analysis team is asked to maximize agreement among coders. Suppose its topic list gives little interpretation and the research question calls for a coherent thematic account. Examine how the team developed the themes. Coder disagreement alone has yet to explain the interpretive weakness.

Develop the themes through interpretation and reflexive discussion, then judge them against the research purpose. If the question requires repeatable fixed-category counts, select and qualify a Method and criteria for that purpose. Agreement among coders can be relevant to the latter task.

RMP.9:5.4 - Correct source extraction without certifying a new tool

An OCR tool repeatedly omits marginal dates from a historical collection. Compare the extraction with the available source images, correct the affected transcription and chronology, and retain the supporting source for the interpretation. The team can retain its historical Method while correcting the extraction.

For a replacement tool and a different handwriting collection, ask whether evidence of extraction fidelity already covers the intended use. If fidelity remains uncertain, compare the tools when access, expertise and effort justify the test. Speed measures throughput. When adequate evidence of the marginal dates is unavailable, qualify the affected part of the argument.

RMP.9:5.5 - Preserve the distinct contributions in mixed research

A project combines fixed-category counts with an interpretive account. A coding revision changes both the counting rule and the meaning of the categories. Reassess the affected count by its measurement criteria and the interpretation by its evidence and purpose, then examine what the two findings can support together.

Use RMP.3, RMP.5 or RMP.7 for the affected operationalization, analysis or synthesis. Retain still-applicable material and inform the users of any claim whose content or basis has changed. Explain how the two contributions bear on the common question.

RMP.9:6 - Bias-Annotation

Numerical examples make some errors and comparison criteria easy to display. Interpretive, historical and community research have other forms of criticism. Use their own professional knowledge to judge an improvement for their purposes.

Novelty and countable gains can attract attention that a suitable practice or a small correction deserves. Saving the cost of a study can also leave an important weakness unexamined. Compare alternatives by the research use, the strength of the evidence and the full cost of obtaining more.

RMP.9:7 - Conformance Checklist

Check the conclusion against the question and conditions it covers.

  • Have you identified the Method, description, implementation, capability or claim at issue, and the research use for which it matters?
  • Does existing evidence settle the question, support retaining the Method or specify a correction?
  • If further evidence is needed, what could it establish, is it attainable, and does its contribution warrant the full effort?
  • If a needed basis is missing, is the limit clear to the person or service considering that use?
  • Does the conclusion distinguish proposed change, actual performance, qualification and observed consequence?
  • Does a changed Method retain the necessary variant, provenance, receiving-condition and recovery distinctions?
  • Do the people or services relying on the result have the information they need about its scope and consequences?

RMP.9:8 - Common Anti-Patterns and How to Avoid Them

Anti-patternWhat failsRepair
Every criticism becomes a new trialThe team starts acquiring evidence despite a determinate correction or an adequate answer.Use sufficient evidence first. For a remaining question, compare the attainable contribution of a new study with its full cost.
No affordable comparison means the Method is adequateAcquisition cost is taken as evidence of suitability.State which uses are supported and where qualification remains open. Restrict an unsupported use when that is within the remit.
Every software change creates a new MethodAn implementation or description correction is mistaken for a change in what the Method requires.Identify what changed and reassess the qualifications that depended on it.
One metric improves every research traditionThe chosen indicator changes the research purpose or overlooks its criticism.Compare alternatives under the actual estimand, interpretation or source-use question.
Only successful trials enter the comparisonFailures and non-convergence disappear from the evidence used for the Method decision.Preserve them and choose their treatment by the comparison’s aim.
New description means successful adoptionAnnouncing a revision is taken as evidence of its use or effects.Establish what support was provided, where the Method was used and what the observations show about its effects, as the claim requires.
Every Method revision restarts the research repertoireIndependent questions, material and claims are reopened together.Trace uses of the changed part and reassess the results whose content or basis depends on it.

RMP.9:9 - Consequences

A team can retain a suitable Method, verify a small correction, or investigate a consequential weakness when attainable evidence justifies the effort. Users can see the conditions supporting the decision and which research results depend on it.

Diagnosis, qualification, introduction and maintenance of a major revision can require specialist capability and substantial work. Include those demands when comparing the alternatives.

Reassess the relevant decision when criticism, instruments, tools, receiving conditions, sources, costs or observed use materially affect its grounds. Trace the particular analyses that rely on the change.

RMP.9:10 - Rationale

Research practice connects a Method difficulty to the standards of criticism for that research and the claims that rely on it. Use Core methods for trials, fit, worth and variants and A.3.1.MI for introduction into practice when one of those contributions is needed.

Identify what needs attention and check the current evidence before commissioning more work. A decision about the value of obtaining evidence can be sound even while a claim about the Method remains unresolved.

RMP.9:11 - SoTA-Echoing

For a question of research-Method improvement, Core Method development (FPF C.39.MF) distinguishes identity, trials, fit, worth and variants. FPF A.3.1.MI addresses introduction into practice. C.11.MV:4.1–4.2 compares attainable courses and their full burdens, including retaining a set of alternatives when the evidence does not distinguish them. Section 4.3 adopts its stopping and acquisition conditions and adapts the comparison to the knowledge question, its standards of criticism and the affected result. In §5.1 the specified denominator makes correction and verification sufficient. If qualification remains open, state which uses lack support. Reconsider further inquiry when the contribution it can supply warrants its cost. Reassess this use of Core if a relied-on condition changes.

For a selected simulation, §§4.3 and 5.2 adopt the common-target design in Morris, White and Crowther (2019), with inspectable planning and reporting from Williams et al. (2024) and treatment of missing or non-convergent outputs from Pawel et al.. Specify the estimand, generating situations, performance criteria and Monte Carlo uncertainty. Investigate failures and interpret them in light of the aim. Comparing only successful runs can obscure unequal failure rates. Include this work in the acquisition cost. These sources guide the conduct and interpretation of a simulation once it has been selected. Reconsider the comparison if the target changes, a new failure appears or better treatment of missing outputs becomes available.

For reflexive thematic analysis, §5.3 adopts the methodological coherence discussed in Braun and Clarke’s RTARG contribution (2024) and their quality guidance. Developing an interpretation and examining it reflexively addresses the weakness of the topic list. Increasing agreement among coders can leave that weakness in place. For repeatable fixed-category counts, choose a Method and criteria for that task. Depending on the coding arrangement, the criterion might concern agreement among coders or the fidelity of an instrument to a reference classification. Judge the quality of the interpretive work from the analysis as well as its reporting.

Section 5.4 adapts the AHA professional statement’s requirements for source integrity and a recoverable trail. Checking the original images grounds a correction to the marginal dates and the chronology that used them. A speed comparison would answer a question about extraction time. A comparison of fidelity for another handwriting collection requires evidence for that collection. Assess available substitutes when access to the originals changes.

The cases apply different standards to statistical, interpretive and historical work. They illustrate how the purpose and cost of an inquiry can change the practical choice. A claim about the effect of using RMP.9 in actual research would require evidence from its use.

RMP.9:12 - Relations

Use A.3.1.MR to recover a candidate way of working from practice. Use RMP.10 when a material reconstruction question needs a larger evidence programme. RMP.11 can explain differentiation among Methods. Use that explanation for a present choice where it bears on the choice, and examine present fit under the relevant conditions.

ContributionActual relation
RMP.1 and RMP.2Use RMP.1 to establish whether a newly opened question qualifies as research and RMP.2 for a needed design comparison or a justified stop.
RMP.3 and RMP.4Use RMP.3 to develop or assess a changed operational choice. Use RMP.4 to establish which records, deviations, transformations and provenance matter for the claim.
RMP.5 and RMP.6RMP.5 examines affected analysis or inference. RMP.6 can establish what an earlier credibility examination showed or help choose a further one.
RMP.7 and RMP.8RMP.7 can reassess an affected synthesis. RMP.8 revises the claim or model and traces the uses of what changed. A difficulty with either Method can enter RMP.9 directly.
Core Method development and A.3.1.MIUse the Core method for identity, repertoire, trial, fit, worth, variants or practice culture when it supplies a needed result. A.3.1.MI supports introduction.
Domain practice and institutional authoritiesUse research evidence in engineering, operational, policy, access or adoption decisions under the criteria and authority governing each decision.

RMP.9:End