Metrics

METRICS

The Food Safety Modernization Act (FSMA) charged the U.S. FDA to develop a comprehensive plan to expand the technical, scientific and regulatory capacity of foreign governments, and their respective food industries, from which the foods are exported to the US.  The plan was published in Feb 2013.  Click here for more information.

Since its inception the Joint Institute for Food Safety and Applied Nutrition (JIFSAN) has conducted a multitude of training and outreach programs in collaboration with the US Food and Drug Administration (FDA), US Department of Agriculture Foreign Agriculture Service (USDA-FAS), National Government Organizations and other partners in promoting best practices in food safety.  In 2012, FDA requested that JIFSAN develop and pilot evaluation tools/instruments to measure the effectiveness and impact of JIFSAN’s international capacity building training programs.

The “Metrics” project established a process to evaluate and improve the impact of JIFSAN’s training programs incorporating process indicators, outcome indicators, and impact indicators which are described below.  (Figure A) below shows how these indicators are interrelated.

EVALUATION METHODOLOGY

Figure A Metrics

 

 

 

 

 

 

PROCESS INDICATORS

  • Pre-training self-assessment – Provides the trainers insight into the group that is about to be trained, such as what they think their skill levels are, and helps the trainers identify the specific training needs for that country/commodity/lab testing method.

 Figure B Metrics

  • Post-training self-assessment – Assesses the participant’s views and satisfaction with the different components of the training, including the instructors’ teaching abilities, and contains a self-assessment of one’s understanding and readiness to implement the different components covered during the training.  Provides instructors insight as to where modifications to trainings might be needed.

Figure C Metrics

  • Factual test (pre and post) – Provides a quantifiable measure on the knowledge gained during the training program. Provides insight to instructors how to alter future training to address questions/issues missed by participants.

Figure D Metrics

Status:  From April 2012 to December 2013 the metrics process indicator tools were administered in 5 Good Agriculture Practice (GAP), 5 Good Aquaculture Practice (GAqP), 3 Food Inspector Training (FIT), 1 Commercial Sterile Food Packaging (CSFP), 6 Microbiology International Training Laboratory (Micro-IFSTL), and 6 Chemistry International Food Safety Laboratory (Chemistry-IFSTL), and 15 Risk Analysis (RA) programs.  Over 1000 participants from around the world participated in at least one of these courses.

OUTCOME INDICATORS

  • Intermediate assessment post training – For JIFSAN’s train-the-trainer programs, the purpose of the intermediate assessment is to measure the multiplier effect (e.g., how many additional training events occurred within a country aimed at the farmer/grower; food processor; food inspector; or laboratory analysts); to understand which components from the training program they were able to use; and if parts were not implemented to understand why.  These are currently being piloted for all the train-the-trainer courses where the process indicator tools were administered. These post training assessments will be administered 6 and 12 months after the training event. In the case of the international Good Agriculture Practice (GAP), Good Aquaculture Practice (GAqP), Food Inspector Training (FIT) and Commercial Sterile Food Packaging (CSFP) programs the evaluation tool is being developed jointly by us and the organization requesting the training and translated into the local language and we dialogue as to the best was to administer the tools such as Google Docs, fax, or email. The partnering organization will then contact participants 6-12 months after the initial training requesting the fill in the follow-up surveys.  In the case of the laboratory training programs the evaluation tool is being developed with the laboratory manager and administered by JIFSAN, through Google Docs.

Status – ongoing; administered to four international training programs attendees by in-country partners; beginning to administer to lab training programs

IMPACT INDICATORS

  • Ultimately we want to be able to assess the long term impacts that the trainings may have contributed to such as increased trade, reduced # of rejected product, reduced food borne outbreaks traced to imported foods, reduced foodborne outbreaks domestically (in the country exporting to the US).  We propose to use existing data sources (Food and Drug Administration (FDA), Centers for Disease Control and Prevention (CDC), United Nations Comtrade (UNComtrade) and we will explore others statistical data bases.

Status – beginning 2014

LINKS

Highlights from AAEA Post conference:

Literature Review under preparation (white paper to be posted shortly)