Deep learning approach to study rodent behavior

During drug discovery and development, systematic safety pharmacology assessments are mandatory prior to first in man clinical trials. These evaluations also include standardized behavioral studies in rodents (rats). An automated video-based system (PhenoTyper^® from Noldus IT1) is routinely used to assess motor behavior during the active phase of the rodents, i.e., during nighttime. This is done using an infrared video camera located in the top unit of each observation arena (=PhenoTyper^®) populated with one rodent per arena (see Figures 1 and 2). During each routine study, a total of 28 rodents are recorded from the top and analyzed for distance moved (see Figure 2) and velocity (not shown) using a “off the shelf” software. As an outcome, the video analysis aims at detecting whether a group of rodents that was exposed to an active ingredient exhibits different behavior, compared to a group of rodents that received a placebo.

• Figure 1: Standardized PhenoTyper^® arena for one rodent
• Figure 2: Screenshot of a video file that typically displays four PhenoTyper^® arenas, housing one rat each, the red line represents the distance track of a rodent of a 2 min of recording

A typical video file will show up to 4 rodents at the same time (see Figure 2). Currently, we analyze and visualize the movement pattern using the Ethovision^® software as shown in the upper left arena of Figure 2 (i.e., distance moved). To relate changes in motor behavior to specific pharmacological interventions, each study includes one placebo (= control) and up to three verum (= active ingredient) groups comprised of 7 rodents each. 14 hours of video material is recorded per rodent (4 groups x 7 rodents x 14 h = 392 h per study).

All studies were performed in Germany, were approved by the responsible German authority and were executed based on standardized protocols. Videos have the following technical specifications:

• Video output: CCIR black/white Vpp-75 Ohm (PAL)  
• 1920 x 1080 pixels
• 25 pictures / sec
• ~ 27 GB / individual video file (as shown in Fig. 2)
• H.264 AVC and MPEG-4 formats

Should you be selected as part of this opn2EXPERTS question, you will have the possibility to access up to 10,000 h of video data for control groups from ~700 individual rodents that were generated as part of 100 individual studies, if required. In addition, 2,000 h of video data for active ingredient groups from ~140 individual rodents were generated as part of 20 individual groups showing altered behavior.

While our current software perfectly detects distance moved and velocity, as part of this call for proposals, we would like to include the detection (i.e., classify and localize in time) of subtle changes of behavior or patterns of behavior from these videos. Comprehensive detection of any behavioral change will ensure safety for future clinical trials.

Given that the control groups can be assumed to display normally occurring rodent behaviors with a certain variability, the above-mentioned video material can be used to train machine learning models to detect potential differences in behavior from treated with active ingredient, e.g., regarding movement, posture, grooming, rearing, sniffing, sleeping, stereotypic behaviors, or other behavioral sequences. Annotated sequences for different behaviors will be provided based on needs specified in your research proposal.

Innovative machine and deep learning approaches based on these video files as an early filter may help to reduce future follow-up studies according to 3R (replacement, reduction, or refinement in animal studies) principles.

In summary, we hope to trigger innovative computer vision approaches based on machine or deep learning for the automated analysis of standardized rodent video sequences to detect potential changes in animal behavior in mandatory standardized preclinical safety studies beyond what is currently provided.

• Computer vision approaches that will be able to analyze video data and be able to identify rodent behaviors as outlined above.
• Machine and deep learning approaches that provide novel insights in our analysis based on supervised learning approaches (i.e., based on exemplary video sequences of typical rodent behaviors) and unsupervised learning approaches (without any descriptions or definitions) are in scope. Unsupervised abnormality detection in the active ingredient rodent groups will be prioritized as part of our scientific review.

The following will be considered out of scope:

• Proposals using commercially available software analyzing animal behavior
• Proposals based on video material from other sources (please note: Only applicable for phase 2 of our call – please see below)
• Classical contract research organization services
• Solution that can only be maintained on external servers as part of a potential final implementation phase (after this call)

The benefits for participating machine learning experts will be different according to the different phases of this call.

Benefits for participating at phase 1 of this opn2EXPERTS call:

You will have the chance to outline and share your expertise, skill set, and prior successes in the field with a top 20 pharmaceutical company, including the potential prospect for a long-term collaboration. As part of phase 1, take the chance to win 5,000 euros as guaranteed honorarium if you were selected for phase 2 of our opn2EXPERTS call. Up to five winners will be selected by our jury comprising of machine learning experts and bioinformaticians to present their unique skill set and ideas on how to tackle our problem statement. Overall, it opens the opportunity to work on a scientific topic in the field of pharmacology.

Your submission of phase 1 will be rapidly evaluated, and we will get back to you in w/c September 19, 2022.

Benefits for participating at phase 2 of this opn2EXPERTS call:

As part of phase 2, benefit from the exclusive access to standardized video output material from our rodent studies which are suitable for novel machine and deep learning solutions as part of a mutual material transfer agreement. We will provide sufficient annotated training video sequences of different animal behaviors from our rodent studies dependent on your needs for the technical implementation and provide additional briefing at the start of the second phase with members of our pharmacology and machine learning / digital teams to train your algorithms. In addition, we will provide sufficient video materials to validate your machine learning approaches based on supervised and unsupervised learning. Each participant of phase 2 will receive the same validation set to ensure fairness for all participants of this second phase. All IP that you use to generate output of your submission of phase 2 will remain with you.

Have a chance to obtain funding of up to 100,000 euros for the overall winning proposal. Your exact funding request should be outlined as part of phase 2 of the call.

The timelines for the second phase will be the following: 
Kick-off (provision of data): w/c September 19 
Submission for phase 2 through November 11 
We strive to finalize internal evaluation and announcement of the overall winner by December 12.

Benefit for the overall winner of this opn2EXPERTS call:

Benefit from funding for an innovative machine or deep learning approach and have the prospect for a long-term collaboration after the end of the delivery for additional future projects with scientists of Boehringer Ingelheim. An agreement for the transfer and use of the program will be established mutually. It will provide full transparency about each partner’s rights & obligations. As part of the agreement, it will be acknowledged that you will be the owner of any potential new intellectual property; however, Boehringer Ingelheim will have a right of first refusal. Furthermore, you will be encouraged to publish following the terms of the collaboration agreement. A possibility to commercialize the final product can be discussed and negotiated. In case a user-friendly, interactive software with easy adjustments to new / different (training) data and video formats is not provided, it could be envisaged as part of a follow-up project together with Boehringer Ingelheim (as part of separate terms beyond this call).

To maintain the highest degree of transparency possible in an open innovation environment, we plan to announce the winner publicly and feature them on opnMe.com and our social media channels.

The key success criteria for participating machine learning experts will be different according to the different phases of this call.

Key success criteria for phase 1 of this opn2EXPERTS call:

• A well-structured proposal containing a new model with a new and compelling approach using our submission template.
• Proven track record of expertise in advanced machine or deep learning analysis applied to computer vision problems.
• Existing approaches that could be easily transferred to the problem statement of phase 2 of this opn2EXPERTS call.
• Include a high-level outline of the minimal set of data that is expected to be required for a successful outcome of phase 2.
• Evidence of prior experience in the field of analyzing any animal behavior via machine or deep learning methods.
• High level documentation on data security of our IT infrastructure to access highly confidential data as part 2 of this opn2EXPERTS call.
• Willingness for long-term cooperation for further development and/or maintenance.
• Proposals that include examples of state-of-the-art object detection, tracking and classification based on public data will be preferred. These can be submitted as part of appendices to the actual word application.
• Please note that your submission should not contain any confidential data at this point.

Key success criteria for phase 2 of this opn2EXPERTS call:

• The project timeline and overall required budget of up to 100,000 euros is structured in milestones and planned with key decision points and budget requirements for each part.
• The proposed concept represents an innovative solution to provide new readouts from existing video tapes.
• The proposed software model based on machine or deep learning can read out the full length of existing video sequences and classify, compare, and differentiate between the potential behavior differences of control and active ingredient cohorts beyond distance moved and velocity as outlined in the background section. Assuming active-ingredient-induced behavioral changes, differentiation of control vs. active ingredient animals and potential identification of dose-dependent effects will be possible.
• The proposed software can analyze a full experiment, comprising of several animals, in groups with different active ingredient treatments whereby each animal will be allocated to a specific group.
• The proposed algorithm can discriminate differences in an unsupervised manner, i.e., able to discriminate beyond what is provided as part of the test sequences. Retrospective tagging of unsupervised observed behavioral differences should be possible.
• The quality of the software should be reliable and robust; sensitivity and accuracy should be demonstrated using the video material we provide.
• The output of the model contains some statistical analysis of differences observed to be able to compare different animal cohorts with each other.
• The final prototype product of phase 2 should be ready to use, and scalable.
• External cloud computing is accepted as part of the phase 2 of the project, but projects that specify that they can be installed on internal Boehringer Ingelheim servers for the actual implementation phase (phase 3) will be preferred.
• Proposals with potential access to source code as part of phase 3 will be prioritized.
• Optionally, the proposed solution provides some sort of graphical user interface to increase user-friendliness. An interactive software with easy adjustments to new / different (training) data and video formats (i.e., changes in the movie acquisition hardware) is preferred, but not required and will not influence final decision.
• Willingness for long-term cooperation for further development and/or maintenance.

Please use our answer submission template to provide a 4-5 page non-confidential proposal (available for download here).

If confidential data exists that would strengthen the proposal, please indicate that information is available to share under a Confidential Disclosure Agreement (CDA). If we find the non-confidential concept proposal sufficiently interesting, we will execute a CDA for confidential discussions.

We are currently seeking answers for the following scientific challenge: Applying innovative computer vision approaches, how would you propose to detect and report subtle behavioral patterns and changes thereof in standardized video recordings of rodents?

All incoming answers accompanied by a collaboration proposal will be evaluated by a scientific jury. Up to five winners will be selected for Phase 2 and awarded 5,000 euros as a guaranteed honorarium to participate in the main call phase. The final winner will have a chance to collaborate with Boehringer Ingelheim scientists and obtain funding of up to 100,000 euros.

We can only accept research proposals if they arrive by the submission deadline on July 07, 2022, 11.59 pm PST.