CONTENT ANALYSIS
We conducted a content analysis of the news published in the most important newspapers, websites, radio, and TV news programs in the participating countries.
Sampling
Researchers in each of the 37 participating countries selected two to four news media outlets per platform, with exceptions in six countries.1 The criteria used to select the media outlets were audience size, reach, and level of influence in agenda setting. Preference was given to outlets that were national in scope, though regional and local outlets were included in the sample in countries in which they were considered important to the media landscape.
Given that the structure and format of media systems differ in many ways across countries including size, audience orientation, ownership, political leaning, and the presence of more than one language in a territory, researchers were asked to ensure as much as possible that the selected outlets represented the diversity of each country’s media system. Team members had to consider the fact that the number of media outlets included may vary from country to country and that greater heterogeneity in the media system would lead researchers to include more outlets, and vice versa.
Thus, for example, certain criteria were used in the selection of outlets: (1) While popular-oriented news outlets do not exist in some countries, they are highly prevalent in others and need to be included in the sample. (2) In some countries, all media outlets are private/commercial, while others have both private and public (mostly broadcast) media. Some of the countries included in the study have only state-run media outlets. The sample should include all types. (3) Researchers attempted to include media outlets that reflect all dominant languages in multilingual countries in which language is an important feature of the media system.
To control for the potential overrepresentation and/or underrepresentation of certain types of media, due to some media including more stories in the sample than others, we weighed the data by medium for each country. This guaranteed that each media type – TV, radio, online news, and newspapers – within each country would have an equivalent weight in the results.
Using the constructed week method, a two-week stratified-systematic sample was selected for all outlets from January 2 to December 31, 2020. The same days were analyzed in all the countries.
Because daily and monthly variations are important factors in studying news content, we divided the year into two six-month periods: January-June and July-December. For each six-month period, we created a constructed week, randomly selecting starting dates on a Monday in January and a Monday in July. Then, using three to four week skip intervals, we selected each of the subsequent six days: a Tuesday, a Wednesday, a Thursday, a Friday, a Saturday, and a Sunday. This procedure allowed us to include seven days in each six-month period for a total sample of 14 days during the year, ensuring that one issue/edition/program from each of the seven days of the week was selected for each half year, and that all months were represented by at least one day.
Some news outlets do not report the news on weekends (Sundays, Saturdays, Fridays) or present the news using formats and/or time slots that differ from those used on weekdays. Thus, in some countries there were no newspapers published or no news programs on television and/or radio on some weekend days. Those cases were considered “missing data.” However, if the time of a television and/or radio news program was changed (e.g., due to sports events), the newscast was coded using the actual time slot.
Each national team determined the specific sampling unit for the selected outlets, based on the following criteria: for television - the most watched newscast within each selected channel; for radio - the news program with the greatest audience in the selected channels; for newspapers - the full issue; and for online news - the entire homepage of the selected websites (including links contained therein).
Whereas television and radio news programs and newspapers are “static” in the sense that each edition is unique and appears at fixed times, online news is dynamic and changes constantly. We therefore “captured” the homepages of the websites at two fixed points during the sampled days: 11:00 a.m. and 11:00 p.m. (local times). This 12-hour gap between the two was assumed to provide the maximum content variability for each day. The homepages and their respective links were opened and saved in real time.
The unit of analysis was the news item, defined as a set of contiguous verbal and, if applicable, audio and/or visual elements that refer to the same event/issue/person. For each of the selected issues/editions/news programs of newspapers, television, radio, and online news websites, all news about the following topics were coded: government/legislatures, campaigns/elections, economy and business, police and crime, court, defense/military/national security, education, health, environment, energy, transportation, housing, accidents and natural disasters, religion and churches, labor and employment, demonstration and protests, social problems, media, sports, science and technology, lifestyle, culture, and entertainment and celebrities.
An event, issue, or statement may be reported by a media outlet in more than one news item on a given day. When the same event, statement, or issue was covered in more than one item, these were considered to be unique stories and were coded separately.
Not included in the study were editorials, opinion columns, weather forecasts, horoscopes, movie (or other cultural) reviews, puzzles, social pages, and similar content on radio and TV. We also excluded supplements/magazines/special features programs and headlines on newspaper front pages and at the beginning of TV and radio newscasts.
Since one of the goals of the project was to compare survey data on journalists’ role conceptions with the content of their news organization, we also excluded content that was not produced by the staff of the respective newsrooms, for example, wire service stories and articles by non-journalists that appeared in news sites; however, content created by freelance journalists was coded. In the case of online media, we only coded news items that appeared on the extended home page by clicking on, and thereby opening, each of the relevant items. Items that included embedded video or audio clips were also coded.
In the end, our sample consisted of a total of 148,474 news stories.
Measurements
The coding was based on the operationalization proposed by Mellado (2015) and validated in previous studies (Mellado et al., 2017; Mellado and van Dalen, 2017; Mellado et al., 2020) to measure professional roles in news content.
The codebook included operational definitions of the performance of the watchdog, citizen-watchdog, interventionist, loyal-facilitator, service, infotainment, and civic roles based on the relationship between journalism and the de facto power, the presence of the journalistic voice in the story, and the way the journalists address the audience. Each of these roles/dimensions is characterized by different measures of professional practices, reporting styles, and narrative schemes.
The codebook used in this wave of the project was refined by the participants in a collaborative manner to ensure intercultural validity. The operationalization of the original indicators, which were designed for the analysis of print media, were adjusted based on the special modalities of radio, television, and online media, including the audio-visual resources of these media platforms, such as sound manipulation, non-verbal expressions, video motion, image frames, and editing (Mellado & Vos, 2017; Hallin & Mellado, 2018). The codebook was applied by all national teams in its original language (English).
Five indicators were used to measure the presence of the “interventionist” role, nine indicators measured the “watchdog” and the “citizen-watchdog” roles, eight indicators measured the “loyal-facilitator” role, five indicators measured the “service” role, five indicators measured the “infotainment” role, and nine indicators measured the “civic” role.
Each indicator was measured on a presence or absence basis. In the case of some indicators, we also coded for the actor or action toward which the journalist’s or source’s comment was directed. We transformed those indicators into dichotomous variables to calculate the main roles and place all measures on the same scale.
Based on previous research, we decided to consider role dimensions as reflective measurement models, where (a) the dimensions exist regardless of the measures used, (b) the variation in item measures does not cause a variation in the construct, and (c) adding/dropping an item does not change the conceptual domain of the construct (Wirth & Kolb, 2012). Additionally, and based on the theoretical rationale of role performance literature (Lynch, 2007; Mellado, Hellmueller & Donsbach 2017), we assumed that journalistic roles could coexist. Therefore, the measures were treated as non-mutually exclusive.
Prior to conducting our main analyses, we completed confirmatory factor analyses to test whether news stories reflected a latent role manifested though concurrent concrete indicators. Within that framework, we empirically tested competing measurement models. CFA results showed a satisfactory fit with the data. Three indicators did not fit well with the data and were excluded from the models: conflict (for the watchdog role), personal assistance (for the service role), and contextual background information (for the civic role). While these indicators were significant for each dimension, their loading was very low. For all roles, we identified each solution as providing a better account of the data than competing solutions.
We also tested for measurement invariance for categorical outcomes, conducting multi-group analyses of measurement invariance. The data show partial measurement invariance across countries.
Based on the CFA results, the individual indicators comprising each dimension were combined to generate a final role score for each item. For descriptive purposes, we calculated raw scores, based on the total points divided by the number of indicators in each role. A higher score expressed a higher presence of each journalistic role in the news, and vice versa. Meanwhile, we used factor scores to test for differences in the performance of the roles analyzed.
The content analysis also included measures related to (1) general information on each news item, such as the type of medium in which it was published, the news outlet, the date of publication, story type, and story placement; (2) the characteristics of the story, such as the topic of the news item and the location where the news story takes place; and (3) the sources cited, including number of sources, source type, the diversity of type of sources, and the diversity of points of view.
Data collection
The sample search process, as well as the news item coding were conducted by native speakers in each country. At least one national principal investigator was responsible for conducting all phases of the study, including liaising with the general project coordinator. National teams were trained by the project coordinator, particularly in order to obtain a common understanding of the operational definitions for all variables of the codebook. Training sessions were conducted during 2019, in-person (the coordinator traveled to some countries) or online.
Coders in each country coded the news stories directly into a specially designed online interface or entered the data manually into SPSS (24 countries used the online interface and 13 countries entered the data directly into SPSS).The corpus of news items in each country was randomly divided among coders to reduce bias and avoid a situation in which a coder would code an entire outlet.
Coding Reliability
Since several concepts are inevitably culturally bound, we followed a three-step strategy to test for intercoder reliability between and within countries. First, we conducted a pre-test among principal investigators across countries to ensure that they had a shared understanding of the codebook. Second, national teams ran pilot tests based on news items not included in the actual sample. As many training sessions were conducted as necessary until coders attained acceptable intercoder reliability coefficients. Coders were also closely monitored during the coding process. During the coding period, each team’s progress was monitored on a monthly basis.
Once the coding was completed, a post-test was conducted in each country, based on 100 items, to measure intercoder reliability. Based on Krippendorff’s alpha (Ka), the final global overall intercoder reliability was .79. The variation in intercoder reliability across roles ranged from .76 to .86, while the variation across countries ranged from .72 to .91.
NOTE
1 In Rwanda, only one radio outlet was included in the final sample. Two were originally included, but one had to be eliminated because it was determined after data collection to be more closely aligned with a talk show than a news program, which violated study criteria. The same situation occurred with the two radio outlets in Qatar and one radio outlet in the UAE. In Germany, only one radio and one print medium were included. While they were able to include a national public radio broadcaster with a large newsroom, there was only one private national radio station with a wide reach at that time, but it did not have its own newsroom. As for the print media, while the aim was to include a quality and a tabloid title, the tabloid with the largest reach was also included in the online sample, so the sampling for the print media focused on the quality newspaper. In Belgium, only one news site was included in the sample. The second news site that met the sampling criteria was part of the same media group as one of the selected newspapers, so the team decided to drop the online outlet to avoid duplication of content.