4  Finding and retrieving data

Data analytics begins before a spreadsheet is opened or a statistical model is selected. The first practical challenge is to determine what data are needed, where those data might exist, and how they can be retrieved in a form suitable for analysis. This is an important shift in perspective. Analytics is not simply a matter of receiving a ready-made file and calculating something from it. Analysts frequently have to translate a business question into data requirements, locate one or more appropriate sources, understand how those sources are structured, and decide how to obtain the relevant observations. 

The search for data should begin with the managerial question rather than with a desire to find an interesting dataset. Suppose a company wants to decide where to open a coffee shop in Nova Scotia. The question immediately suggests several kinds of evidence: population and population growth, household income, employment patterns, business locations, commuting patterns, and perhaps information about existing competitors. No single source is guaranteed to contain everything required. The analyst may therefore need to combine internal organizational data with government statistics, open data, commercial or research data, and other sources. 

This chapter introduces some categories of data sources and practical strategies for finding and retrieving them. It also emphasizes a distinction that will recur throughout analytics: finding a source about the right topic is not the same as finding data with the right observations, variables, geography, time period, and level of detail. Successful retrieval begins with a carefully defined need. 

4.1 Start with the business question

Before searching the web or opening an organizational database, identify the decision or question that the data are expected to support. A broad question such as “Why are sales declining?” needs to be translated into potential measures. Useful internal variables might include transactions, product categories, prices, promotions, customer segments, store locations, returns, or inventory availability. External data might include population change, inflation, weather, competitor activity, or broader economic indicators. The particular list depends on the business context. 

This translation produces search concepts. A question about whether population growth is associated with housing prices in Halifax might require housing prices, population, geography, and a time variable. Those concepts are more useful for data discovery than typing the entire explanatory question into a web search engine. Searching for “Halifax housing prices dataset,” “Nova Scotia housing prices CSV,” or “Halifax population Statistics Canada” is more likely to surface downloadable data than asking a search engine why prices are increasing. 

It is also important to determine what is the intended unit of observation. Will one row represent a customer, transaction, business, community, province, month, or year? This decision helps determine whether a candidate source has sufficient granularity. A provincial annual population series may be authoritative and accurate but still be irrelevant when the decision requires neighbourhood-level comparisons. 

4.2 Example types of data sources

4.2.1 Internal organizational data

When working for a company, public agency, nonprofit organization, or client, internal systems are often the first place to look. Organizations generate information through routine activities: customer relationship management systems, point-of-sale transactions, inventory systems, human-resource platforms, accounting systems, websites, mobile applications, service records, and operational databases. These data can be especially valuable because they are closely connected to the organization’s own decisions and processes. 

Internal data might describe clients, products, inventories, sales, employees, expenditures, financial performance, or interactions with services. However, “internal database” should not be imagined as a single clean table containing everything an analyst wants. Different organizational systems may have been created for different operational purposes. Customer identifiers may differ across systems; dates and categories may be encoded differently; access may be restricted; and data definitions may reflect administrative needs rather than analytical ones. 

The analyst should begin by understanding what systems exist, who is responsible for them, what each record represents, and what permissions are required. Retrieval may occur through a report export, a CSV or Excel download, a database query, a business-intelligence interface, or an API. The appropriate method depends on the organization and the system. Regardless of method, you should retrieve only the information needed for the defined purpose and preserve documentation about the source and extraction.

4.2.2 Open government data

Governments and public organizations make substantial amounts of information available through open-data programs. Canada’s Open Government site provides a search interface for information resources and datasets published by government institutions. The federal catalogue includes resources distributed in multiple formats, including CSV, XLSX and JSON on some records. Open government data can support questions involving the economy, public spending, regulation, transportation, the environment, businesses, communities, and many other management-relevant topics. 

At the provincial level, the Nova Scotia Government Open Data Portal is designed to make government data easier to access for individuals, businesses, and researchers. Its catalogue organizes datasets across subjects such as business and economy, government administration, social services, and nature and environment.

Statistics Canada deserves particular attention because it offers several retrieval mechanisms. Its developer documentation states that users can download complete data tables in CSV format with accompanying data and metadata, while its Web Data Service provides programmatic access to aggregate data and metadata. Statistics Canada also provides JSON and SDMX-based services for various uses. This means the same source can support a beginner downloading a table for Excel and a more advanced analyst building a reproducible workflow that retrieves selected observations through an API. 

Open data should not be confused with data that can be used without reading any conditions. Analysts should check the applicable licence, attribution requirements, and documentation. For example, the Nova Scotia Open Government Licence permits broad reuse subject to conditions including source acknowledgement and identifies exclusions such as personal information and third-party rights. Licensing is therefore part of responsible data retrieval, not an afterthought. 

4.2.3 International and municipal public data

Management questions often cross national boundaries. The World Bankprovides an important source for international development and economic indicators, while national and municipal governments maintain their own portals. In the United States, Data.gov acts as a federal open-data catalogue. At the local level, cities, public libraries, transit organizations, and other public bodies may publish datasets that are more granular than national statistics. 

The level of government or institution should match the question. National indicators are useful for country comparisons, but local business-location decisions may require municipal or neighbourhood data. Similarly, a public library may publish information about branches, usage, programs, or collections that supports questions about public services. The important lesson is that “government data” is not one source. It is an ecosystem of producers operating at different geographic and administrative levels. 

4.2.4 Research data repositories

Research data provide another important source. Open research practices encourage researchers to preserve and share datasets and other research outputs so that they can be discovered and reused. DataCite supports discovery by connecting research outputs and resources through DOI metadata. DataCite Commons provides an integrated interface for searching and filtering metadata, complemented by machine-actionable services such as a REST API and OAI-PMH. 

General-purpose repositories are useful when data are not housed in a discipline-specific archive. Zenodo describes itself as a general-purpose research repository in which users can share and preserve research outputs including data, software, publications, and presentations. Zenodo also recommends considering discipline-specific repositories where appropriate. Figshare similarly supports storing, sharing, and discovering research outputs and explicitly presents itself as a repository for data and other scholarly materials. 

Research repositories can be especially useful when a journal article refers to a dataset that would be useful for a new analysis. However, it should not be assumed that publication in a repository automatically makes a dataset relevant or methodologically appropriate. The repository helps with discovery, preservation, metadata, and citation; the analyst still has to inspect the dataset’s documentation, methods, variables, licensing, and suitability for the new question. 

4.2.5 Miscellaneous public datasets and dataset search engines

The web contains many additional datasets. Kagglehosts datasets alongside analytics and machine-learning activities. The Awesome Public Datasets project on GitHub curates links across many topics. IMDb makes downloadable datasets available for non-commercial use through its interfaces page. Such sources can be valuable for practice, demonstrations, and particular research questions, but their authority, licensing, documentation, and provenance vary. A well-known hosting platform should not be mistaken for the original producer of every file hosted there. 

Google Dataset Search offers another discovery strategy. Instead of searching only ordinary webpages. This can is useful when the analyst knows the topic but not the repository. A dataset search engine is therefore complementary to curated library guides and dedicated government portals: the guide points to selected sources, the portal searches a known collection, and the search engine searches across many sources. 

Search strategy still matters. It can be helpful to combine substantive keywords with terms such as “dataset,” “data,” “CSV,” “XLSX,” “JSON,” “download,” “API,” or “open data”, or search within a known domain when they want a specific producer. The goal is to move from webpages discussing a phenomenon to records that expose the observations themselves or explain how to retrieve them. 

4.2.6 Library data guides and human expertise 

A web search is not always the fastest route. University libraries often provide data and statistics guides that organize major sources by subject, geography, or publisher. The Data and Statistics Guide from Dalhousie Libraries can serve as a curated starting point for locating statistical resources. Guides of this kind reduce the discovery problem by connecting users to credible repositories and explaining what kinds of data they contain. 

The value of a guide is not that it replaces searching. Rather, it narrows the search space. Investigating where to locate a coffee shop might begin with census and population resources, then identify business or trade data, and then supplement them with provincial or municipal sources. A librarian or data specialist can also help clarify whether a desired dataset exists, whether access is licensed, and how a particular interface works. 

4.3 Recognizing the actual data 

One of the most common difficulties for beginning analysts is distinguishing a webpage about data from the data themselves. A news release, chart, dashboard, report, or landing page may summarize information without exposing the underlying observations in a form that can be analyzed.

File formats provide clues about how retrieval will work. CSV is a simple tabular text format that can be opened in spreadsheet applications. XLSX is an Excel workbook and may contain multiple sheets, formatting, formulas, or metadata. JSON is a structured text format frequently used by APIs and can contain nested objects and lists rather than a flat table. API access allows software to request selected records or fields programmatically. More specialized sources may also offer XML, SDMX, GIS formats, or database access. 

Format choice should reflect the task. A small one-time analysis may be easiest with a CSV or Excel download. A project that must update regularly may benefit from an API. A large table may be better retrieved selectively rather than downloaded in full. The analyst should also obtain metadata or data dictionaries whenever available, because variables without definitions can be difficult to interpret correctly. 

4.4 From downloading to querying: methods of retrieval

Downloading a file is the most visible form of retrieval, but there are several common methods.

The first is direct file download, often as CSV, XLSX, JSON, or a compressed archive.

The second is database querying. In a relational database, a query can select particular columns, filter rows, join tables, and aggregate results. Even managers who do not become database specialists benefit from understanding that a query is a precise request for a subset of stored data. 

The third method is an API. An API allows an application to request information from a service in a structured way. Statistics Canada, for example, describes its Web Data Service as a mechanism for accessing aggregate data and metadata directly from its public databases and documents JSON-based methods. API retrieval can make an analysis reproducible because the selection logic can be recorded in code, but it also requires attention to documentation, authentication, usage conditions, pagination, and changing service requirements. 

The fourth method is specialized search or export software. Some systems provide built-in exports; other services are accessed through packages in R or Python. The general principle is to choose the simplest method that reliably retrieves the observations needed while preserving enough metadata to understand them.

4.5 A note on data collection ethics

While this chapter mainly focused on finding and retrieving data that is already collected and made available to us through some medium. Sometimes we will collect our own data from surveys or other tracking mechanisms. This section discusses some ethical considerations related to data collection.

Data collection is never simply a technical activity. Decisions about what information to collect, from whom, for what purpose, and for how long can affect the people represented in the data. For managers and analysts, ethical data collection therefore requires more than asking whether information is available or useful. It requires considering whether collecting the information is necessary, proportionate, transparent, fair, and respectful of the people from whom the data originate.

Privacy laws establish important legal requirements, but ethical practice should not be reduced to legal compliance. Something may be technically possible, or even legally permissible in a particular circumstance, without necessarily being a good or responsible analytical choice. An ethical analyst should therefore ask two related questions:

  • Are we permitted to collect these data?
  • Should we collect these data?

The second question requires professional judgment.

4.5.1 Purpose and necessity

A good starting point is to identify the purpose of the collection. Why does the organization need the information? What business, administrative, or research question will the data help answer?

Canadian privacy principles provide a useful framework. The Office of the Privacy Commissioner of Canada explains that, under PIPEDA, organizations must identify the purposes for which personal information is collected before or at the time of collection. Collection should then be limited to information needed for those identified purposes.

This leads to the principle of data minimization: Collect the data you need, not simply the data you can get.

Suppose a gym wants to understand why members cancel their memberships. Variables such as membership type, membership fee, number of visits, classes attended, cancellation date, and stated reason for cancellation have an obvious connection to the business question. Other information might already exist in organizational systems, such as credit-card numbers, emergency contacts, health information, or precise location histories. The fact that these variables are available does not establish that they are necessary.

The Office of the Privacy Commissioner of Canada advises organizations to limit both the amount and type of personal information collected to what is needed for the identified purpose. It notes that collecting less information can also reduce the potential consequences of loss or inappropriate access, use, or disclosure.

Data minimization therefore has both an ethical and practical rationale. Every additional variable collected must be stored, managed, protected, and eventually disposed of.

4.5.3 Sensitive information

Not all personal information presents the same level of concern. Some information may have particularly significant consequences if exposed, misinterpreted, or used inappropriately. Depending on the context, examples could include financial, health, location, biometric, employment, or identification information.

An analyst should therefore ask:

  • Do we actually need information at the individual level?
  • Could the question be answered with aggregated data?
  • Could we remove direct identifiers?
  • Could we use broader categories rather than highly detailed information?
  • What would happen if this information were disclosed or used for another purpose?

Removing names is helpful, but analysts should not assume that removing obvious identifiers eliminates all privacy concerns. Canadian federal guidance on de-identification notes that information may still constitute personal information where there is a reasonable expectation that individuals could be identified from the information alone or in combination with other available information. It therefore advises consideration of the context and risk of re-identification.

The Office of the Privacy Commissioner of Canada similarly notes that de-identification and related approaches can reduce privacy risks when appropriately applied, but definitions and legal consequences vary between jurisdictions.

4.5.4 Public does not mean ethically unrestricted

The distinction between accessible and appropriate is especially important when dealing with information found online.

Suppose Harbour Fitness discovers that many members have public social-media profiles. A marketing analyst might argue that the information is public, so why shouldn’t we collect it?”

Ethically, this is the wrong starting point. The relevant question is not simply whether the analyst can access the information. The analyst should ask why it is necessary for the defined business purpose, whether collecting it would be reasonably expected, and whether a less intrusive source could answer the same question.

The same principle applies to automated web collection. Large quantities of information can sometimes be gathered because technical tools make collection easy. Technical scalability does not eliminate ethical responsibility.

4.5.5 Fairness and potential harm

Ethical data collection should also consider how information might affect people after collection. Variables selected for analysis can influence how customers, employees, communities, or other groups are categorized and treated.

The Office of the Privacy Commissioner of Canada states that purposes considered inappropriate can include profiling or categorizing individuals in ways that lead to unfair, unethical, or discriminatory treatment contrary to human rights law, as well as collection, use, or disclosure for purposes known or likely to cause significant harm.

For a management analyst, this suggests looking beyond immediate analytical usefulness. Ask:

  • Could this variable result in people being categorized unfairly?
  • Are we using a proxy for something we should not be using?
  • Could the dataset systematically underrepresent some people?
  • Could a prediction be interpreted more confidently than the underlying data justify?
  • Who benefits from this data collection, and who assumes the risks?

These questions become particularly important when analytics informs decisions that affect people.

4.5.6 Responsibility continues after collection

Ethical obligations do not end once information has entered a database. Collection creates responsibilities related to storage, access, security, retention, and eventual disposal.

Under PIPEDA’s fair information principles, personal information must be protected with safeguards appropriate to its sensitivity. The principles also limit use, disclosure, and retention to identified purposes unless another basis applies.

This means an analyst should consider data lifecycle questions before collecting the data:

  • Where will the information be stored?
  • Who genuinely needs access?
  • How will access be controlled?
  • How long does the organization need the information?
  • What happens when the information is no longer needed?
  • If information is shared with another analyst or organization, what protections remain in place?

Collecting information creates an obligation to manage it responsibly.

4.5.7 Ethics as part of analytics

Ethical collection should ultimately become part of the normal analytical workflow rather than a separate compliance exercise performed after decisions have already been made.

Before adding a variable to a dataset, an analyst can ask five simple questions:

  1. Purpose: What specific question requires this information?

  2. Necessity: Could we answer the question without collecting it?

  3. Transparency: Would the person represented in the data reasonably understand this use?

  4. Risk: What could happen if the information were exposed, misinterpreted, or used differently?

  5. Alternatives: Could less sensitive, less detailed, aggregated, or de-identified information accomplish the same objective?

The goal is not to discourage organizations from using data. Data can improve services, reveal patterns, support better decisions, and help organizations understand their operations. Rather, the goal is to recognize that more data are not automatically better data. For managers, perhaps the most useful ethical principle is therefore the simplest: Do not begin by asking, “What data can we collect?” Begin by asking, “What data do we actually need, and can we justify collecting them?”

That change in perspective places ethics at the beginning of the analytical process, where decisions about data collection are actually being made, rather than at the end, when the information has already been collected.