Author: Belen Egozcue

Welcome back! In our last blog post, we used AWS QuickSight’s basic features to analyze the gender gap in the tech industry. In this post, we will take a closer look at the newer QuickSight technologies designed to make data analysis simpler and more effective. We will explore QuickSight Q, Forecasts and Insights, and uncover the practical applications of each. Delve into the power of data analysis with us and let’s get started!

QUICKSIGHT Q 

What is Quicksight Q?

First and foremost, QuickSight Q is a natural language query (NLQ) tool used for BI analysis that allows users to ask business-related questions and receive the most representative visualizations as answers. What sets QuickSight Q apart from any other business intelligence tool is its capability to provide data-driven insights – in response to analytical queries – through the leverage of machine learning technologies. 

QuickSight Q strives to reduce the need for BI expertise in order to analyze the data, achieving this by providing the most accurate visuals based on its interpretation of natural language. With its intuitive data analysis skills, Q is sure to revolutionize the way companies interact with their data. 

Now that we have discussed the theoretical aspects of Q, let’s dive into the more practical applications of the technology, its most important features and how they can help us visualize and comprehend our data.  

How does Q work? A practical look into it

By far the most important feature to understand when getting started with QuickSight Q is “Topics”. As defined by Zac Woodall at AWS re:Invent, topics are a collection of data that align with a subject area that your business users can ask questions about (1). In simpler terms, these are the data Q utilizes in order to answer the given questions. We will be using the dataset from our first blog post about the gender gap in the tech industry to examine how we can create a topic and kickstart our QuickSight Q journey!

In order to create a topic that enables us to ask questions about Kaggle’s surveys, QuickSight offers two alternatives. The first option involves creating the topic from scratch, linking it to an already existing QuickSight dataset. While the second option entails creating a topic directly from an analysis we want to derive new insights from. Let’s see what the latter looks like.  

Create and Edit New Topic From Analysis

When the Enterprise + Q subscription in QuickSight is enabled, automatically, all of your analyses will include an upper search bar that will allow you to ask questions about your data. Before we can do this though, we need to “Enable a topic” with the current dataset used in the analysis.

After clicking on “Enable topic”, a window should pop-up where we will be able to name our topic and explain what information can be found in it. QuickSight Q by default selects the dataset used in the analysis, so in case of wanting to ask questions about a different dataset, then you will need to create a new topic separately based on that data.

Once we have completed that step, QuickSight Q will start indexing and configuring the analysis’ data. But you might be wondering, what exactly is indexing? This is the process by which an AI system looks into and prepares your data for natural language processing. What QuickSight does is examine every feature in the data – from the name of each column to the data type of these – and renames the associated columns to make them more understandable, while simultaneously identifying synonyms users might use to refer to the data. 

After the indexing process is completed, the search bar will have the topic enabled, allowing users to start making questions. However, before doing so, it is important to review the topic configuration to ensure that Q’s performance is optimized and that users receive the best possible answers to their questions.

Inside the topic configurations, we can make alterations to help Q draw conclusions from data. For example, we can include or exclude attributes from which Q will generate visuals, as well as adding or removing synonyms to enhance QuickSight Q’s understanding of what we want to visualize. It is important to keep in mind that the synonyms and field names should be unique since Q uses these in order to link questions with them. By ensuring these settings are correct, we can help maximize Q’s performance and accuracy of responses.

Moreover, we have the option to modify the data type and add details like the default aggregation and semantic type in each data field. QuickSight Q already creates these configurations automatically, however, we must review them to ensure each field is classified correctly. If we don’t, Q could omit key elements from the data, consequently affecting the responses. 

Finally, it is worth mentioning the “User Activity” tab, where QuickSight Q displays a quick evaluation of its answers, for instance how many of them we thought answered our question. 

Let’s get Q-rious!

Having outlined the functionality of Q, let me provide some examples of how it can be used to create quick analysis. Say we wanted to visualize the number of respondents throughout time and by gender. One option would be to just ask Q “show gender by date” and that would give us the answer we are looking for, like portrayed in the image below.

However, what would happen if another user asks the same question but with slightly different wording? For instance, “show respondents by year”.  Given the field synonyms that QuickSight Q identified, that information cannot be answered. This is what is meant when saying it is the author’s responsibility to review questions users ask and modify the data labels to help Q provide better answers. 

So if we want to improve results, we should add some synonyms to the list and try if by doing that Q recognises what we want. 

By adding the tags “year”, “years” for the field Date; “respondents”, “women”, and more for gender, we are telling QuickSight Q to use said fields when a question contains those words. This should help us get the results we want. 

As evidenced in the plot above, by incorporating new words to the analysis, QuickSight Q returns a visual that shows how we can obtain results that better adhere to our natural language. However, a table is not the most effective way to illustrate the number of respondents over time. Fortunately, QuickSight Q enables users to switch to a more suitable visual type that meets their requirements. 

Simultaneously, by clicking on the “Mark as reviewed” button, when new users ask questions about our dataset, they will be able to observe how the question was approved by the author. So next time, instead of displaying the initial visual in table form, Q will have learnt that the desired visual type is an area graph.

“Why” feature

Another of Q’s most impressive features is its ability to provide analytical explanations as to why certain things are occurring in your data. For example, in the last graph, we could see how the number of women respondents increased in 2022, but it may not be immediately apparent why this happened. By writing the prompt “why did women respondents increase in 2022?” QuickSight Q offers us an overview of the different fields in our dataset which contributed the most to this increase. 

As depicted in the image, Q identified that the total number of women had risen 8% between 2021 and 2022. Moreover, Q provides further insight into the fields that are considered to be “key drivers” for this change. For example, the number of female respondents that were from India increased by 32%, being deemed responsible for 132% of the increase of female surveyees. This aligns with what was discussed in the last blog post, regarding the ongoing growth of the tech industry, particularly in India.

It also informs us that women in the 40-44 demographic increased by 29%, contributing a total of 23% to the overall increase. These insights are similar to those we reached in our first blog post, just by analyzing the data. This demonstrates how we can reach similar conclusions, and understand why our data is behaving in a certain way without BI expertise, simply by asking Q.

We could go on and on about QuickSight Q’s remarkable features, but the ones mentioned above are the ones that, personally, stood out the most. These features provide the insights and understanding necessary to get the most out of your data so you can analyze it like a pro in no time!

QuickSight Forecasts & Insights

An Introduction to Forecasts and Insights

Moving onto other Machine Learning-powered QuickSight technologies, we can discover Insights and generate Forecasts. The first tool helps us uncover hidden outliers and insights that may be difficult to identify at first glance in our data. On the other hand, AWS QuickSight Forecasts enables users to predict future values based on historical data, by utilizing sophisticated algorithms that generate those forecasts. 

Similarly to QuickSight Q’s purpose, QuickSight Forecasts and Insights are designed to be used even without prior Machine Learning expertise, reducing the reliance on analysts to understand and draw conclusions from our data. By making these tools accessible and easy to use, users can quickly identify patterns and phenomena in their data, and make more informed decisions.

In order to gain a better understanding of these QuickSight technologies, let’s once again look at some practical examples of what one can create, and how these tools can help us comprehend our data. 

Quick~Insights!

If you want to get powerful insights from QuickSight, it is important to first create graphs and allow the tool to start analyzing your filtered data and draw conclusions from the plots. These do not need to be the fanciest of plots – just select a few features and you are good to go! It is worth keeping in mind, though, that filtered data will provide more specific insights, rather than generic ones. 

In order to access the auto-generated insights, we should look at the left toolbar and click on Insights. 

Once the Insights tab is selected, QuickSight will show multiple widgets that contain data insights that have been gathered from the fields used in our graph. Take the following image as an example. 

The “Suggested Insights” QuickSight provides contain a comprehensive overview of our first plot, which displays the gender distribution in 2022. Although the narratives are auto-generated, QuickSight allows users to customize them to meet their specific needs and ways of displaying the information. 

Since QuickSight generates these insights in widget form, we can include them in our analysis at any given moment, and modify the narrative if necessary. Simply click the “+” button to add them to the sheet and click on the “Customize narrative” to make changes if desired. 

With just a few clicks, QuickSight helps users get the most out of their data by providing valuable information such as the top values, percentage changes over time, highest and lowest values, and much more. This is particularly useful for quickly analyzing the key details of a feature – for example calculating the year-to-date comparison between two years – which would otherwise require a considerable amount of time from a BI analyst. 

Overall, QuickSight lives up to its name, since it is an extremely useful tool for anyone looking to get quick insights from their data! Its user-friendly interface makes data analysis more accessible, helping users kickstart and inspire their analyses in just a matter of seconds.

QuickSight Forecasts

Last but not least, QuickSight offers the option to add forecasts to our graphs. With the help of an in-built Machine Learning algorithm, QuickSight allows users to forecast how their data will behave in the future. 

All we need in order to forecast our data is a date field and the metrics that we want to evaluate. In this case, we will be forecasting the gender wage gap in the United States to try and predict, based on past data, whether this gap will decrease in the future. (4)

So to add a forecast to our plot, we have to click on the option “Add forecast” and automatically its properties will open up.

Through these parameters, we can fine-tune our prediction by changing the periods forward we want to forecast, the periods backwards we want to use to determine patterns in the data , and the confidence level used to build the prediction interval. QuickSight by default detects the seasonality in the data automatically, however this can be changed if another time period better reflects the seasonal pattern found in the data. Finally, boundaries can be added to the forecast values if necessary, ensuring that they do not surpass these limits. 

What is also interesting is how QuickSight provides extra information for each parameter, so even if you are unfamiliar with what seasonality is, you can get an understanding of what is expected by hovering over the information bubble next to the parameter name.

Conclusion

Undoubtedly, QuickSight technologies such as Q, Insights and Forecasts are incredibly powerful tools that can provide organizations with valuable insights, and forecasting capabilities. By utilizing these, organizations can gain valuable knowledge on their businesses in more efficient ways and with little prior knowledge on the technical aspects of the tools. With the right implementation QuickSight can help organizations stay ahead of the curve and make better decisions. So, take the leap and go beyond the limits of data with QuickSight!

Interested in exploring the use of Quicksight to draw business insights? As AWS Advanced Partners our team at Montevideo Labs has extensive experience with AWS services at scale. Contact our team to learn how we can help you in your cloud journey! 

References

AWS Events. (2022, December 2). AWS re:Invent 2022 – Get clarity on your data in seconds with Amazon  QuickSight Q (BSI207). YouTube. Retrieved March 24, 2023, from https://www.youtube.com/watch?v=BkjUSd8JK9c 

Earnings and wages – Gender wage gap. (n.d.). OECD Data. Retrieved March 24, 2023, from https://data.oecd.org/earnwage/gender-wage-gap.htm#indicator-chart 

Forecasting and creating what-if scenarios with Amazon QuickSight – Amazon QuickSight. (n.d.). AWS Documentation. Retrieved March 24, 2023, from https://docs.aws.amazon.com/quicksight/latest/user/forecasts-and-whatifs.html 

Working with insights in Amazon QuickSight – Amazon QuickSight. (n.d.). AWS Documentation. Retrieved March 24, 2023, from https://docs.aws.amazon.com/quicksight/latest/user/computational-insights.html

It goes without saying that technology has drastically changed our world – from the way we communicate, work and study, to the way we spend our free time and access information. Ever since the global pandemic, technology has found even more ways to establish itself in our lives. However, despite the industry’s undoubted growth and impact on society, there is still one big issue that often goes unnoticed: the gender gap. In the tech industry, women are severely underrepresented, which not only is detrimental to their own professional aspirations, but also to companies’ potential for innovation and creativity. 

Being a Data Science student myself, I find it extremely important to shed light on the issue of gender disparity within the tech industry. To do so, I will analyze the annual Machine Learning and Data Science surveys provided by Kaggle (Kaggle, 2022), using AWS QuickSight – a highly powerful business intelligence service –  in order to gain a better understanding on how this situation has evolved in recent years and if we can expect it to change for the better in a near future. 

Data preparation

Kaggle’s annual industry-wide survey is a key element for understanding the state of the tech world. By taking an in-depth look at the survey data of the last four years (2019-2022), we can gain valuable insights into how the gender gap has evolved in the industry. The surveys cover a wide range of topics related to the sector, gathering answers from thousands of people from all parts of the world who offer a useful outlook on how the industry works. It is worth mentioning how Kaggle’s surveys are primarily filled by Data Scientists, Analysts and Engineers, given its renown in the data sector. Although the surveys are not limited to any particular tech area, these professionals are more likely to respond due to the website’s reputation. 

To accurately analyze the gender gap using Kaggle’s surveys, I had to first identify which questions I believed were the most relevant to the analysis. As the questions were so varied, I decided to focus on those where the gender disparity could be clearly detected, such as age, salary and education.  Once I had selected those, I proceeded to the data preparation phase, where I concatenated all the survey datasets for each year into a single dataset in order to begin plotting the results. Quicksight has an in-built data preparation section, which allowed me to change the data types of the attributes and create new features based on the ones already uploaded. These proved to be highly useful actions when preparing the data. 

The infamous gender gap

Now, the biggest question we all might be asking ourselves is: how bad really is the gender gap in the tech world? Let’s break it down into the fields: gender, age, country, salary, highest education and work role, in order to understand its prevalence in society better.

A look into the gender distribution

Out of the 24 thousand respondents, 18.270 were male (76%) and 5.290 were female (22%), representing an approximate 3:1 ratio of male to female respondents. Less than 1% of the respondents identified as non-binary or not provided an answer.

As discouraging as the gender gap in 2022 looks, in comparison to the last 4 years, things clearly have been looking up since 2019 . 

Each year displays a small but still significant increase in the number of women in the field. In 2022, there was a 3.25% increase since 2021 and an overall 5.84% increase since 2019.  Even though this might be a sign that more opportunities are gradually being offered to women, the disparity still remains. This is why it is beyond important to raise awareness on the matter, so that newer generations of women can grow without limitations and achieve everything they set their minds to!

Now we will analyze the origins from which participants come from, in order to see if any countries in particular are offering more opportunities to women in tech. 

By far, India and the United States of America show the highest numbers of answers. India’s female and male answers amounted to 58% and 54% respectively, whilst the United States’ answers amounted to 20% women and 17% men. It is intriguing to note how, regardless of gender, India and the US have the highest percentages of professionals from the tech industry among the survey respondents.

Of the remaining countries on the continents of South America, Africa and Asia each contribute 5% or less of female and male answers. Even though these countries have a lower female response rate, it is important to consider that the socio-economic and cultural context, as well as the resources to participate in the survey or even be a part of the tech world, vary greatly among these.  Thus, identifying countries where female participation is more likely to increase is essential.  

As evidenced in the plot above, the number of respondents in India, Nigeria and the USA has increased over the years, making it worthwhile to analyze the industry’s development in some of these countries. For instance, the information and communication technology (ICT) sector in Nigeria contributed 18.44% to the country’s GDP in the second quarter of 2022, showing a 6.55% growth from the first quarter (NAN, 2022). To further encourage female participation in the ICT sector, Nigeria has implemented the Girls in ICT program which provides mentorship, training and career guidance for young women (United Nations, 2016). Similarly, India’s tech industry is expected to grow 8.4% in 2023 after the sector attained revenues of around $226 billion in FY2022 (Bagchi, 2023). Additionally, WITI (Women in Technology International) is an Indian organization set to help in the inclusion of women in business and technology (WITI, n.d.). These statements help to demonstrate the potential for more inclusion of women in the tech industry.

Age is just a number – or is it?

Age is also an important factor when it comes to analyzing the gender gap.

Quicksight’s useful feature called “small multiples” allows us to visualize and compare the differences in age distribution between genders across the years. For instance, when examining the female age distribution, the majority of female surveyees fall into the 18-21, 22-24 and 25-29 clusters. Interestingly, since 2019, the number of women between the ages of 18-21 has increased significantly, indicating how younger women are starting to get more involved in the industry as the years progress. 

Despite both genders’ age distributions being right skewed, with the majority of respondents being between the ages of 18-40, the male distribution reveals a disproportionate amount of male respondents over the age of 40 in comparison to females respondents. This disparity could suggest how, historically, the gender bias was even stronger, and it was not as welcomed for women back then to pursue technological or mathematical careers. 

As it would be expected, younger women are prevalent in the tech industry throughout the top 10 countries with the highest number of female answers. 

From the graph displayed above, we can observe that nearly 50% of women are paid an annual salary of less than $10.000, while only 36% of men earn less than this amount. 

Although the gender wage gap persists, we can observe an encouraging trend in the data from the last two years. The percentage of women earning less than $10,000 annually has decreased, indicating that women in the industry are being paid more for their hard work. This is a positive sign that progress is being made towards closing the pay gap.

Furthermore, it might be enlightening to explore what educational backgrounds are associated with higher-paying job opportunities in the tech sector for women.

In 2022, women with a Master’s degree dominated each salary range, with those holding a Bachelor’s degree or Doctoral degree following behind.

Though possessing a Master’s degree might not guarantee a six figure job, it could potentially increase the probabilities of achieving higher wages. What’s more, women who lack any formal education beyond high school – although rare cases – can still earn an annual salary of over $125.000, demonstrating that although higher levels of education can open more doors, the tech world is not limited to those with the most prestigious educational backgrounds. 

Finally, let’s analyze the most frequent work roles in the last year, and see what differences gender has on them.  

The survey results for 2022 show that the most popular work roles across both genders are Data Scientist, Data Analyst, Software Engineer, Teacher/ professor and Research Scientist, respectively. Although these professions are in high demand nowadays, we should be mindful of possible biases in the survey results,  considering that Kaggle is a site primarily used by data scientists/analysts. 

While the percentage difference between genders is not significant, some insights can be derived from them. For instance, it can be observed that the female to male respondents working as teachers or professors ratio is approximately 2:1. Additionally, the results showed that 3% more women work as Data Analysts than men. 

Overall, female Data Scientists have the most potential to earn the highest salaries when compared to other tech roles. Data Analyst and Software Engineer follow closely behind, with these roles also having a significant presence in highly rewarding salary ranges.  

Though future salary expectations play a key part when it comes to choosing a career path in tech, the opportunities that come with these jobs are endless, allowing for women to pursue their passions and make a real difference in the world.

Conclusion 

To conclude, I would like to share a famous quote from Malala Yousafzai, a Pakistani female activist who fought for women’s rights. This quote, taken from one of her speeches at the United Nations in 2013 (Yousafzai, 2013), I believe captures the essence of this blog flawlessly.

“We call upon all communities to be tolerant — to reject prejudice based on cast, creed, sect, religion or gender. To ensure freedom and equality for women so that they can flourish. We cannot all succeed when half of us are held back.” – Malala Yousafzai

A person’s gender should never be a limitation or indicator of what one can amount to. It is essential to abandon any stereotypes or prejudice we might have in order to succeed as a society, because as Malala pointed out: “We cannot all succeed when half of us are held back.”

The gender gap is an immensely complex issue that deserves further exploration. In this post, we have analyzed the past and the current state of affairs. In our next post, we will continue our analysis using advanced AWS Quicksight technologies, such as Quicksight Q, and we will try to get a glimpse of the future, using QuickSight’s Forecasting features, so stay tuned for more! 

References

Bagchi, S. (2023, March 1). Indian tech industry to grow 8.4% to $245 billion in FY23: Nasscom. Mint. Retrieved March 7, 2023.

Kaggle. (2022, October 16). 2022 Kaggle Machine Learning & Data Science Survey. Kaggle. Retrieved March 7, 2023.

NAN. (2022, August 27). ICT sectors contribute 18.44% to Nigeria’s GDP, says NBS. The Guardian Nigeria News. Retrieved March 7, 2023.

United Nations. (2016, 04). “Girls in ICT Day” highlights need to promote technology careers for women. the United Nations. Retrieved March 7, 2023.

WITI. (n.d.). About – WITI. Women in Technology International. Retrieved March 7, 2023.

Yousafzai, M. (2013, July 12). Malala Fund. Malala Fund. Retrieved March 7, 2023.