Background:
This case study will showcase Q Analysts Data Collection expertise and infrastructure, including our ability to recruit over 300 participants with diverse demographics to provide our client with the high-quality data they needed for human audio data. The focus of the project was bias testing for a new and upcoming health and fitness band device. The client approached us for this project because of our expertise in human data capture and our ISO27001 compliant Q TestLab facility. At our state-of-the-art Q TestLab, we can control the collection and ensure the security of prototype capture devices.

Challenge:
The client requested bias testing from our participants’ verbal tones and emotions when speaking, ensuring that its new device performs for people of all ages, ethnicities, and regions. The problem was insufficient data – the data collected from the client’s internal team was not very diverse. The client also experienced a secondary issue in which its data was very monotone or neutral in emotion because its internal data was collected exclusively in a professional office setting.

The client presented its requirements and worked with our Ground Truth Data Services team to find the best possible strategy to execute. The primary focus was to collect as much conversational data as possible while still making participants feel comfortable. We proposed a tandem collection process where two participants would be in the same room (in the same COVID bubble) to double the amount of data collected. We worked collaboratively with the client to make sure we thoroughly understood their needs. The anonymity of the client and the security of the devices were our top priorities. From there we leveraged our Participant Recruitment Team to project velocity and timelines.

Solution:
The planning period took a couple of months due to pauses and delays caused by the COVID-19 pandemic. Once the collection study was approved, we were able to begin right away with one week of pilot and training. Recruiting had begun pipelining participants two weeks before the start of the project. We staged real-life furnished rooms in our Kirkland Q Testlab facility where individuals would participate in the capture. Each capture was about four hours in length. Individuals participated by wearing the band device while it recorded their conversations. Participants could talk about anything except any personally identifiable information.

Results:
While we successfully performed the collection, the devices were incredibly unstable, with frequent crashes and data upload failures. This led to multiple two-week long pauses during the project to allow the client to investigate and resolve the data upload issues. To mitigate the impact of these issues, our team performed controlled tests with the client to investigate the cause of the upload issues. This meant performing test captures in various scenarios which all had to be documented in detail (time, device IDs, recording times, recording length, etc.) to help determine possible causes. Data collection of this nature is an ongoing and collaborative effort. While this did not resolve the upload issue completely, most of the data did finally upload, giving the client enough varied data to make its devices more inclusive!