Category Archives for "Market"

Regression Algorithms Used In Data Mining

Regression Algorithms Used In Data Mining

Regression algorithms are a subset of machine learning, used to model dependencies and relationships between inputted data and their expected outcomes to anticipate the results of the new data.

Regression algorithms predict the output values based on input features from the data fed in the system. The algorithms build models on the features of training data and using the model to predict value for new data.

They have many uses, most notably in the financial industry, where they are applied to discover new trends and looking at future forecasts.

Here are five of the most used regression algorithms and models used in data mining.

Linear Regression Model

Simple linear regression lets data scientists analyse two separate pieces of data and the relationships between them.

The model assumes a linear relationship exists between input variables and the singular output. This can be calculated from a linear combination of both the input variables.

Examples of linear regression models include predicting the value of houses in the real estate market and analysing road patterns to predict where the highest volume of traffic is.

Simple variable regression implies that there is only a single input variable. Where there are more than one variables, the method is known as multiple linear regression.

Unlike linear regression technique, multiple regression is a broader class of regressions that encompasses linear and nonlinear regressions with multiple explanatory variables.

One business application of the multiple regression algorithm is its use in the insurance industry to decide whether or not a claim is valid and needs to be paid out. This example has many different variables to consider, so a linear regression algorithm wouldn’t be appropriate,

Multivariate Regression Algorithm

This technique is used when there is more than one predictor variable in a multivariate regression model and the model is called a multivariate multiple regression. It’s one of the simplest regression models used by data scientists.

Multivariate regression algorithms are used to predict the response variable for a set of explanatory variables. This regression technique can be implemented efficiently with the help of matrix operations.

These algorithms are used as part of the AI revolution of the medical industry. Doctors require a lot of data collected from their patients, including heart rates and cholesterol levels to external factors such as how much they exercise.

They may want to investigate the relationships between their patient’s activity compared to how much cholesterol they have in the body.

Logistic Regression

This next data mining regression algorithm is another popular method used in the financial industry, particularly in the credit checking business. This is because logistic regression requires a binary response.

In regression analysis, logistic regression estimates the parameters of a logistic model. More formally, a logistic model is one where the log-odds of the probability of an event is a linear combination of independent or predictor variables.

There are two possible dependent variable values: “0” and “1”. These are used to represent outcomes such as pass/fail or win/lose.

One of the major upsides is of this popular algorithm is that one can include more than one dependent variable which can be continuous or dichotomous. The other major advantage of this supervised machine learning algorithm is that it provides a quantified value to measure the strength of association according to the rest of the variables.

Going back to the credit scoring industry, it can be easy to see how it is used; companies apply logistic regression to see if a customer meets the necessary criteria to be eligible for a loan/credit card/etc.

Lasso Regression

Lasso (ie Least Absolute Selection Shrinkage Operator) regression algorithms are used to obtain the subset of predictors that minimize prediction error for a quantitative response variable. The algorithm operates by imposing a constraint on the model parameters that causes regression coefficients for some variables to shrink toward a zero.

If the algorithm assigns a coefficient to zero, the variable is no longer used as part of the model. Those that have a non-zero coefficient are then used as part of the response.

Explanatory variables can be either quantitative, categorical or both. Lasso regression analysis is basically a shrinkage and variable selection method to determine which of the predictors are most important.

Lasso regression algorithms have been widely used in financial networks and economics. In finance, the models have used stock market forecasting, such as how it will react to economic updates and predicting where to invest or the stocks and shares to stay away from.

Support Vector Machines

The final regression data mining algorithm is the support vector machine (SVM). This machine learning regression model is a supervised learning model with associated learning algorithms to analyse data used for classification.

Support vector machine algorithms build models that assign new examples to one category or the other, making it a non-probabilistic binary linear classifier.

The model is a representation of the examples as points in space, mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible. New examples are then mapped into that same space and predicted to belong to a category based on which side of the gap they fall.

In addition to performing linear classification, SVMs can efficiently perform a non-linear classification using the kernel trick, implicitly mapping their inputs into high-dimensional feature spaces.

Support vector machine regression algorithms have found several applications, including recognition systems that can detect whether an image contains a face. If the SVM identifies a face, it produces a box around it. Data visualisation techniques are included within the SVM algorithm.

How can AI help the problems of wealth distribution

AI in relation to wealth distribution

The gap between the rich and the poor has been steadily growing but with the advancements in AI and automation, is it likely that this gap will get even wider?

One of the main benefits of using AI is that tasks can be completed faster and more efficiently than ever before.  Businesses are lapping it up because it saves valuable time and money.

However, it is feared that what may seem a great thing may actually further disconnect the wealthy and everyone else.

Advancements in artificial intelligence mean that jobs are being replaced by intelligent systems.  This has actually been in place for a number of years now, with machines replacing workers in factories on the assembly line for example, but obviously, now things are much more advanced.

Automation has been hitting manual labour jobs hard because employers can save a lot of employment costs and ensuring the jobs are doing with minimal errors.

The difference nowadays is that the machines are getting smarter.  Smart technologies are everywhere, from TVs to security systems to kettles.  Technology is starting to eliminate even the most basic of tasks such as boiling water.

There is a history of advancements

AI will continue to develop to where it will replace jobs that do not require much training in order to function necessary tasks.

But this is not anything new.

Human civilisation has always found a way to improve efficiency.

Take the industrial revolution for example.  Machines were implemented to improve production output.

A number of inventions were made that saw the improvement in the textile industry and the rise steam powered engines.

Modern history is no different.

The invention of the internet has seen people switch from doing everyday things in person to doing them online.

One of the big benefactors of this has been online retailers.  The normal shopping experience of visiting the local supermarket or high street is being replaced with internet spending.

Why?  Because it is so much easier.

Doing things from your own home is less stressful and takes a lot less time.

AI on the workforce

The main issues that surround this is fear of the unknown.  Will AI end up costing everyone their job and only the rich can survive?

Highly unlikely.

While it may be likely that automation will end up replacing a lot of untrained and lower skill-based jobs, people will be able to learn an entirely new skill.

Automation systems that have machine learning abilities will not be able to replace everything a human can do.  It can be trained to think like a human but it won’t be on the same level.

Think about it from the perspective of Kallum Pickering, an analyst at Berenberg:

Producers will only automate if doing so is profitable. For profit to occur, producers need a market to sell to in the first place.

Keeping this in mind helps to highlight the critical flaw of the argument: if robots replaced all workers, thereby creating mass unemployment, to whom would the producers sell?

Because demand is infinite whereas supply is scarce, the displaced workers always have the opportunity to find fresh employment to produce something that satisfies demand elsewhere.

As you can see, automation will have to increase the number of jobs in order for companies that create and develop the technologies to sell their product.

The more jobs created, the fewer unemployed and so the gap between the rich and poor should close.

The flip side

What is really interesting is that in the UK, the rate of unemployment has been falling over the last number of years.

However, wages do not seem to be rising as fast as they should be.

Is this because of automation?  Perhaps.

With the rise in machines being used to replace jobs, the ex-workers have to find some other form of employment.

AI is strong in logic and complex thinking but struggles with basic tasks.  For example, a computer will be able to easily solve difficult mathematical equations but won’t fare as well as a postman without being embedded into a movement device.

It most likely pays less to deliver letters but because a computer can’t do it, the unemployed have little option but to take these jobs.

So if this were to happen, it could be said that AI will increase economic inequality because jobs will cease to pay well based on the low skills that are required to do them.

The less money in circulation, the less wealth can be distributed.

How to manage the spread of wealth

The more that technology replaces humans in the workplace, efficiency will increase which in turn, increases wealth.

The current problem we have is that when we start to see the benefits, they should be available to everyone.

This is the major point that needs to be tackled to solve the wealth distribution problem.

The fewer people that are in work, the more the government will have to support the unemployed with extra welfare programmes.

At the same time, the jobs that are created must reflect economic sustainability in order for workers to maintain a worthwhile lifestyle.

There are schemes that are being trialled now in order to prepare for the robots age such as the universal basic income.  This has been rolled out in Finland and is being said to make people want to go out and get a job.

But this is just one potential route that can be taken and governments across the globe will have had discussions about how to keep up with advancements.

There must be a genuine attempt at sorting this out so we are not left behind.

As Larry Elliott describes:

Inequality, without a sustained attempt at the redistribution of income, wealth and opportunity, will increase.


If used properly, integrating AI into modern life will only help society advance to levels we have never experienced before.

However, using too much too soon could put too many people out of work and increasing the gap.  It is important that those that could be threatened by the integration are educated on how they can stay in work but not have to settle being worse off because of a robot.

What Is The Captcha Library & How Does It’s Existence Fuel The AI Revolution

What Is The Captcha Library & How Does It’s Existence Fuel The AI Revolution?

A lot of websites, whether you’re signing up or purchasing something, are now asking you to prove you are not a robot.

How many times have you seen the image on the right and being asked to answer a question to show you’re human?

It’s everywhere; CAPTCHA has taken over and is becoming a major part of the AI revolution.

But what is it?

Let’s start a the beginning.

What is CAPTCHA?

A Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA) is a challenge-response test used to tell human and robotic behaviour apart.

The most common type of CAPTCHA was first invented in 1997 which required users to distinguish the type of letters in a sentence.  The test was that the images were usually distorted in some way as a computer would find it difficult to read.

The tests are carried out by computers rather than humans.  This often leads to CAPTCHA tests being referred to as reverse Turing tests.

A lot of web applications are now implementing CAPTCHA as part of their on-screen security measures.

Properties of CAPTCHA tests

CAPTCHAs are fully automated computer operations that require little to no human input and maintenance.  This means that businesses can save a lot of time and money on resources and maintain consistency.  Once programmed, CAPTCHA programmes provide accurate and reliable tests.

The algorithm used to create the CAPTCHA must be made available to the public, though it may be covered legally by a patent. This is because breaking the CAPTCHA programme requires the solution to a difficult problem in the field of artificial intelligence (AI). 

Modern text-based CAPTCHAs are designed such that they require the following abilities to be used at the same time:

Invariant recognition

Invariant recognition refers to the ability to see the numerous amounts of different possibilities to how a shape can look or presented.  A computer needs to be taught how to successfully identify these possibilities as a human has an edge; the brain.  Teaching these to a computer is actually incredibly difficult.


Segmentation is another power of CAPTCHA.  This enables the programme how to separate and distinguish one letter from another.   Again, this is another challenging task for CAPTCHAs the letters are usually clustered together with no white space between.

Unlike computers, are very good at distinguishing patterns.  Computers have to separate the recognition and segmentation processes whereas humans compute both at the same time.  The human brain combines both into the same process.


Context is the final skill but is just as important as the previous two.  CAPTCHA must be understood to correctly identify each character in the given phrase. For example, in one segment of a CAPTCHA, a letter might look like an ‘o’.  However, after reading the word and understanding the context, it becomes clear that the letter is actually an ‘a’.

Humans are able to automatically understand what context the given text is applied.  By being able to do this, humans cannot be tricked into thinking one letter is another.

On their own, each of the three above challenges is a tough task for a computer to complete.  All three at the same is ridiculously hard.  This is what drives the consistency that CAPTCHA provides a computer system.


Most CAPTCHAs are used for security reasons; as we saw at the beginning of the article, the reCAPTCHA check is being used by numerous businesses.

However, CAPTCHAs are also a standard for AI technologies.  As said by von Ahn, Blum and Langford in their article on using hard AI problems for security:

Any program that has high success over a CAPTCHA can be used to solve an unsolved Artificial Intelligence (AI) problem

A difficult example of an AI problem is that of speech recognition.   CAPTCHA programmes may use this technique as the underlying method for identifying human v robotic interaction.

von Ahn, Blum and Langford go on to say in the article that as CAPTCHA is used for security purposes, it is important that the AI problems that use it are useful.

If the AI problem is useful, there is either a way to differentiate between computers and humans, or a useful AI problem has been solved.

Each time a CAPTCHA is solved, the computer is taught how to do it again.  This is a machine learning technique; each time the computer is taught the solution, it becomes more accurate.

But where did they get these from?

Original CAPTCHA strings were actually scans of complicated words from old books that existing computers couldn’t recognise. The original developers wanted to use the CAPTCHA system to finally convert some of the oldest works in existence into digital format. In this process, they had found that traditional scanning methods such as OCR could not detect certain words. Naturally, to go through these words manually would take them an infinite amount of time due to the sheer size of the number of books they were keen to convert! Thus, using these indetectable words as their dataset on the original CAPTCHA system was not only a safe solution to the captcha problem (as they could be sure computers hadn’t recognised the text), but also they could use this to get a lot of people working on this at once to help convert the books!

What Is Home Automation & Is It Secure?

What Is Home Automation & Is It Secure?

Smart technology is booming.  Smart homes ie home automation is becoming increasingly popular with people investing more and more into automated technology.

It’s getting to a point where everything can seemingly can be automated and controlled by an app.

This post will be discussing what home automation is and whether or not it is safe to use.  After all, anything that is smart can technically be hacked… including your kettle!

What is home automation?

Also referred to as domotics, home automation is, as the name suggests, building automation for a home.  Machine learning and data mining play an important role in most home automation systems.  It helps to predict user activity.

The process involves hooking up the device with internet in order to be activated.  The modern systems are controlled by mobile phones, computers, tablets or off-site systems that have access to the internet.

Although not usually linked with home automation, early forms include the invention of the washing machines, refrigerators, dishwashers and tumble dryers.

Nowadays, most home automated systems are being controlled by apps.   Examples include:


This is the area where most smart home technology is being developed.

Home security applications like the Nest are one form of home smart security applications, ranging from cameras to alarms.

Some of the applications can be programmed to recognise the homeowners face so if there is an intruder and the camera picks them up, the technology will know that the person is not a friendly face.

Updates can be sent to the user’s mobile phone via the application to give them all necessary notifications in real time.

Light and heat

Hive are a company that specialise in providing smart heating and lighting to someone’s home.

The user simply attaches the hub to their boiler or heating system which is then controlled remotely by the user’s phone.  They can then set the remotely temperature to their desire.

The same process works for the lighting.  You simply insert a light bulb into the socket and then control it via the mobile phone app.

Home entertainment

I’m sure you’ve heard of the term smart TV by now.  It’s the latest buzzword all over the adverts.

This means that a television can be connected up to the internet meaning that a user can browse the web, on demand catch up TV and access online streaming services like Netflix.

The software is loaded into the TV’s operating system and available through the application, similar to accessing apps on a smart phone.

This same techonolgy can also be found on games consoles, DVD and Blu-ray players.

Kitchen appliances

Kitchens are getting increasingly ‘smarter’.  Technology is advancing so that ordinary appliances can be connected to the internet to provide an even easier use.

Refrigerators are becoming automated.  Smart refrigerators let a user see the contents of their fridge on their mobile device.

Home automation can also be used by radios or television units by connecting them to the home WiFi.

You can now even buy smart kettles.

A user can control the exact temperature they want their water to reach when boiling.

All from the convenience of a mobile phone application.

Of course all these examples have one thing in common: they are all intended to make life that much easier to things you do every day.

Even the kettle!

With all this technology however, there are a lot of concerns.

Is home automation safe?

There are a few common questions that crop up:

Can all this technology be hacked?  Will it work when I need it the most?  Will it turn on by itself?

Considering a lot of smart technology that is being developed is being used in the security business, it’s pretty important that the answers to these questions can put to bed any fears people may have.

To sum it up as simply as possible: home automation is safe.

Connecting all your security equipment and other appliances up to the internet is safe and secure.  The data that is shared between your device and the internet is encrypted so even if there is a breach, an intruder can’t do anything with it.

As long as your internet passwords are strong, it is unlikely that anyone can get access to your network in the first place.

Stay secure

Here are a few tips to keep in mind if you have or looking to installing any smart technology in your home:

  • When setting up your new or existing devices, change the default password to something memorable that only you would know.
  • Use a different password for each different device and keep them on a need-to-know basis.  If you only use a single password for everything and there is a intrusion, the hacker can access everything.
  • Include a combination of at least an uppercase letter, lowercase letter and number in each password.
  • If you haven’t got one already, make sure your phone has a password or some sort of security on the lock screen.  All your apps can be accessed on this one device so keep it secured.
  • Install all updates that are available.  Most updates will always include some sort of security upgrade to keep you as safe as possible.

All these are simple to follow and easy to implement straight away.

Keeping your network and devices is the most important thing to do when integrating automation into a home.

Once you have done all these, you will be secure and enjoy the many benefits of smart technology in your household.

A smart home is a very efficient way of living in modern society.  They are very convenient and make usually easy tasks, such as flicking a light switch, effortless.

Just bear in mind how many devices you really need to connect to your network.

Do you really need that kettle to tell you the exact temperature your water is at?  Pick only the equipment and appliances you really need to access the internet and you will be less vulnerable to potential breaches.