Practice, Ethics & Integrity

1. Documentation

• Critical for 2 things:

  1. Objective replication

     • Hypothesis

     • Design

     • Prediction

     • Procedures

     • Instruments

     • Contacting Participants

  2. Checking and verification of results and conclusions

     • Data
     • Data manipulation
     • Statistics
     • Interpretation

• Available On line

  Instruments & materials

  the instruments used to measure or manipulate the variables of interest, and the procedures or instruments used to control or measure extraneous variables.

  • written or verbal instructions

  • text used in email communications

  • consent forms

  • documents containing debriefing information

  • Instruments

    1. how many questions a scale in a questionnaire
    2. what the response options are
    3. what the range of possible scale scores is
    4. an example item is often provided

    

  • research protocol 实验流程方案

    the order and manner in which materials or information were presented to the participants and the way procedures were implemented.

    • the moment of first contact
    • how a participant is recruited with what information and by whom.
    • how consent is obtained
    • what instruction or information is given, in what order
    • what help is provided if participants don’t understand instructions or they behave in an unexpected manner.
    • how participants are debriefed about the research after their participation.

  • preregistered 备案

  • pilot study

2. Data Management

1. Store

   • code book: information or metadata about what the variables mean.

     • what property each variable measures
     • what values mean
     • what the range of possible values is
     • missing value: what values are used to denote if participant did not supply relevant information

     

   • original data

     • paper questionnaires filled out by participants
     • video material used for observational coding

   • duplicate files

2. Process

   • Recording

   • Computing sum scores

3. Analyze

   • Syntax file 语法文件

     a simply programming file that can be used to reproduce all the computations and statistical analyses at the push of a button.

     • checking and replicating results

3. Unethical studies

• Tuskegee syphilis study 塔斯基吉梅毒研究
• Guatemala study 危地马拉研究

4. Ethics towards participants

• Institutional Review Boards (IRBs) 伦理审查委员会

• Three Principles:

  1. Respect 尊重

     • Respect for the participant’s autonomy.

       • Voluntary consent vs. Coercion
         • extremely large financial reward
         • special treatment to diseases
         • students for course credit

     • Consent form

     • Often, some form of deception is necessary to control for reactivity and demand characteristics.

       • Omission: the goal of the study is not stated or formulated in very general vague terms.

       • Active deception: a cover story.

       • False feedback

         • !!danger!!: perseverance effect 固化效应

           participants are still affected by the deception even after they’re debriefed and the deception is revealed and explained to them.

  2. Beneficence 善行

     • the participants should not be harmed.
       • cost benefit analysis
         • individual level
         • broader level
       • invasion of participant’s privacy
         • solution: anonymity of data should only be promised if identifying information is deleted.
         • who will have access?
         • what it will be used for?

  3. Justice 公正

     • cost and benefits of research should be divided reasonably, fairly and equally over potential participants.

5. Research Integrity

• Openness, transparency and critical and systematic empirical testing.

• Fraud

  • data were either fabricated or falsified to provide false support for the researcher’s hypothesis.

    • Fabrication 伪造

      • Data were never collected, but were made up.

    • Falsification 篡改

      • existing data were illegitimately altered

• Plagiarism

  • a substantial scientific contribution is presented as one’s own by copying original text, concepts or data of others without referring to the original source.
    • new type of plagiarism—self-plagiarism 自我抄袭

• Conflicts of Interest

  • researches are funded by parties of interest.

• Personal Values

  • strong convictions or personal values can blind researchers to their data and valid critiques.

6. Questionable Research Practices

• Practices that are acceptable if they’re implemented objectively and responsibly. But they can be abused to obtain more favorable results.

  

  • Harking 事后假设

    • hypothesizing after the results are known
      • hypothesis is adapted to fit the observed data.
    • a priori hypothesis can be considered relatively strong support for a hypothesis
      • Prediction is much harder.
    • a posteriori hypothesis formed after the fact.
      • it’s easy to find an explanation that fits a specific result.
      • the original hypothesis is not confirmed.

  • P-Hacking P值操纵

    • data manipulation or selection that makes the results(p-value) more favorable.
    • The golden rule of p-hacking is that as long as data selection and manipulation are reported and arguments are provided, the reader can judge for himself whether these choices are justified.

    

    • Cherry-picking 摘樱桃

      • reporting only results that are favorable and significant.

    • Selective Omission 选择性遗漏

      • the omission of non-significant results and results that contradict the hypothesis.

    • Data snooping 数据探测

      • the collection of data exactly until results show a favorable p-value.
      • sample size should be determined beforehand based on non-arbitrary estimates of the expected effect size of a treatment and the required confidence level

• as long as choices are reported, they should be discussed and their influence on results can be evaluated.

7. Peer Review Process

8 Dissemination Problems

• Side effect: publication bias

  • favoring the publication of new and exciting significant results.

• file-drawer problem

  • it’s more rewarding to do a new study in hope for positive results and leave the null results study in a file drawer.

• solutions:

  • preregistration of the research question, the design and statistical plan.
    • eliminate harking, cherry picking and selective omission.
  • basing the decision to accept or reject a manuscript on preregistered research proposals instead of completed studies.
    • eliminate publication bias and the file drawer problem.
  • the data and statistical analysis are made publicly available.
    • eliminate p-harking, fraud