Data Scrubbing 101

Data Scrubbing 101

Team Polyrific

Data is always at the core of every business and process, but data is not always perfect, and it can take time and effort to properly identify when something has gone wrong.

                We have all seen inconsistent, repeated, or often misspelled information that can affect the business by complicating human interaction or automated processes running in the background, generating additional expenses while reducing productivity and efficiency.

                Data scrubbing has the main objective of preventing data from becoming unreadable or unusable in the broader sense; this is achieved by the implementation of continuously running processes in the background such as:

  • Consolidation of data (removing duplicate or redundant information)
  • Correction of inconsistencies (ensuring that all data in the respective fields or locations follow a set standard as established by the business)
  • Data integrity (ensuring no data corruption or data is missing)

Achieving effective data scrubbing can be a long and arduous task but one that comes with several benefits for the business; primarily, this must be achieved through agreement from all departments involved to ensure data is consistent and useful.

Once this is achieved, the implementation on the back end is performed to ensure data is available to all parties through the agreed standard; this may also include information from several databases in different
formats or with different pieces of information.

Integration and implementation of other tools in the data scrubbing process can achieve additional benefits such as verification of data, such as addresses or other personal information when possible, to ensure that these match with data that is publicly available.

Note that the process of data scrubbing can be performed at any level, coming from data that is being ingested, to information at rest in a file, database, or other formats in which it may be available or in its final location, although it is always preferred to be done before it reaches its destination for ease of handling.

Is your data scrubbing process working for you, or are you looking to implement such a process in your business?

We can help you with this and more; contact us for a free evaluation of your current needs, and we will help you build a work plan for the implementation of a better data scrubbing process that ensures your data is really speaking to you.

Team Polyrific | Mar 10, 2021

As working in the Cloud becomes more inevitable with every passing moment for business looking to grow, some features and services can benefit your business with considerable savings and ease implementation and maintenance times.

    One such service is available through Azure with SQL PaaS (Platform as a Service), which takes all the heavy lifting out of managing a database, letting you focus directly on your data and the applications that interact with it.

    But what about everything else that is involved with handling a database? You do not have to worry about anything else; Microsoft has you covered on this front so you can concentrate on what matters.

    This includes patching and updating the back-end, additionally assisting in identifying, reporting, and taking actions on any potential threats to your business data and databases.

    Azure SQL PaaS also allows you to grow as much as you need when you need it, at a fraction of the cost from your on-prem and so large it can easily handle big-data without breaking the bank.

    The added advantage to this model means that your data will always be available with minimal to no down-time for your applications to work at full speed, wherever you are, cutting the need to synchronize multiple databases around your business units.

    One key feature for Azure SQL PaaS is the ability to use it to have it as a redundant backup for your in-house on-prem applications when moving them to the Cloud is not always feasible.

    There is also the possibility of migrating all of your data through tools provided directly for Microsoft. There is no need to build special tools or perform complicated migrations that can take a large amount of manpower, money, and months to be completed.

    Microsoft also ensures that backups are performed continuously, creating a robust redundancy to assist in rapid recoveries and storing them for extended periods in case you need to roll-back through a considerable period of time.

    Being part of the Azure services, there is an integration made available with the Azure AD, allowing for Single Sign On (SSO) and faster account handling, avoiding creating additional accounts or handling multiple user databases on-prem.

    Are you interested in moving to the Cloud and moving your database information to Azure SQL PaaS?

Contact us through the provided method below, and we will get back to you to assist with all your migration and integration needs.

Team Polyrific | Jul 09, 2018

Despite the ever increasing pace of advancements in the software world, there are still no out-of-the-box solutions that work perfectly for every enterprise. If you think about it, this is actually a good thing because if your enterprise operates in exactly the same manner as your competition, then there is nothing to competitively separate you from them and that is one of many reasons that we aren't fans of one-size-fits-all solutions that are critical to your core business operations.

If you have lived through or are in the midst of rolling out a big name software package, we'll wager that it has caused you your fair share of heartburn. Here are ten reasons why your off-the-shelf implementation isn't going so well:

1. You are paying for features you don't need and won't use

Take a look at any off-the-shelf software solution and you are likely to find myriad features that have no relevance to your business. This makes sense because the software in question was undoubtedly built to "work" for the widest possible market segment and this means inclusion of features that may make sense for others but not for you. The problem is, your license fee is covering the cost of those features you will never use. A pervasive example of this is the reporting features that get bundled with most off-the-shelf ERPs & CRMs. It's likely that most of those reports aren't being used at your enterprise.

2. Hidden costs

Off-the-shelf rollouts (especially ERP and CRM) are never as easy or fast as the well-funded salespeople told you they'd be. Many ERP rollouts take between 6-12 months which is time that you are paying consultants to make this "out-of-the-box" solution do what you paid a license fee to have it do. Think of the irony there for a moment--shouldn't an "out-of-the-box" solution just work right out of the box?

Apart from the high cost of implementation there is also the cost of penetration/security testing, upgrade charges, the cost of extension packages that make the solution behave in a way that better works for your business, recurring annual licenses, and the cost to train your staff to use the solution that may not operate in a manner they are used to.

3. The software doesn't fit the business

No off-the-shelf solution will perfectly match the way your organization does business. At best, it can only come close. This means that your stakeholders are going to have to shape their business practices around the way that the software operates.  

A dangerous aspect of off-the-shelf solutions is the temptation to underinvest in requirements gathering and business analysis because you figure that the "off-the-shelf" software you just paid a mint for will cover their needs. This creates resentment among your stakeholders who feel they are not being heard and it stymies good ideas that they may have brought to the table for boosting productivity, innovation, and profits.

4. "Configuration" means customization

Think that the "off-the-shelf" approach will mean that your aren't doing any customization, think again. In the COTS (commercial off-the-shelf) world the word "configuration" might as well be "customization" and it will cost you just as much or more than a solid custom solution would have in the first place.

5. Lack of free and fast support

There are going to be times that you encounter an edge case bug in your software or you simply have made a mistake such as deleting data that you need to retrieve. In the "off-the-shelf" world you will either have to pay for fast tier 3 support or you will have to wait--sometimes for weeks--for the vendor to resolve your problem. Also, since the software's source code is closed to your team, you are not empowered to have your in-house engineering staff tackle the problem for you meaning that their hands are tied even though they may have been able to resolve the issue quickly.

6. No control over the product roadmap

Have a great idea for a feature that would be hugely valuable in your off-the-shelf software package? The vendor might be willing to hear about it and, if so might be willing to implement it *if* they think it is in line with their wider market sector. Even if they do decide to move forward with developing the feature it could be months or even years before you see it.

7. No competitive separation

Think of the industry in which you and your competitors work as a gene pool. When you are all using the same big name off-the-shelf products, there is a great likelihood that you are all being channeled into doing business in the exact same way. You can probably see where I am going with the gene pool analogy: lack of diversity in the way you and your competitors do business means lack of evolution and competitive separation.

Competitive separation is further limited through the limits that off-the-shelf solutions place on innovation. Very few in your organization are going to come forward with innovative ideas for bettering your business if they know they would be impossible to implement within the context of your off-the-shelf solution. What would be the point?

8. Lack of usability leads to lack of adoption

Can you imagine foisting an out-of-the box ERP interface on a field mechanic who has to log his or her equipment inspections? It happens everyday though--golved or greasy hands having to manipulate a stylus to fill out forms better displayed on a desktop in a cubicle. Swearing ensues and sometime tablets grow wings.  Your field personnel are good at what they do and they do not want to spend too much time living in your world. A much better solution for them might have been a custom solution offering a voice interface that they can run on their smartphone. Both Google and Microsoft offer APIs that make this an easy feature to incorporate into custom applications but you don't have that ability with your off-the-shelf solution.

9. Lack of integration with existing software and data

There are off-the-shelf products that now ship with integration capabilities, often for other big name off-the-shelf products, but many still offer no such integration and none of them have a way to interface with your enterprises native data sources. For that to work, you need yet another custom engineering effort, often in the form of "glueware" built to tie your new off-the-shelf product to your existing data stores. 

There is an alternative--new off-the-shelf "integration platforms" help you tie all of your packages together but require high amounts of implementation time and consulting. At some point, it becomes easier to just build a more focused custom application in the first place.

10. High total cost of ownership (TCO)

One thing that will certainly put a sour taste in your mouth is the increasing level of cost of your application over time. Thought you were done spending money when you wrote the initial licensing check? Not even close. Before you have used your expensive off-the-shelf product for a few years you will likely have spent money on annual licensing fees, upgrades, security testing, market place extensions, support, and so on. You also might have to allocate funds for internal staff to monitor and maintain the off-the-shelf product so that you can be sure it stays up-to-date and compliant. Besure to create a five-year total cost of ownership projection when considering off-the-shelf software solutions.


As you can probably tell, we aren't big fans of "off-the-shelf" solutions at Polyrific because in the best case we think they are agents of averageness and in the worst case we think they negatively impact the clients who we care about. It's not to say that all off-the-shelf solutions are bad of course, but you do need to carefully consider your decision before going in that direction. Don't be afraid to ask around--other folks you know might have had a run-in with the software you are considering and can give you the real story. We don't know too many people who are happy with their off-the-shelf purchase decisions but we know of plenty who are delighted with our custom solutions!

If you are interested in developing a custom application to better your enterprise, please feel free to contact us for help. We are experts in developing world class custom enterprise solutions.

Team Polyrific | Feb 06, 2018

If you haven't already heard the term "NoOps" as it pertains to enterprise software development and delivery you probably will soon. NoOps is an emerging movement that seeks to relieve a bottleneck created by traditional IT operations and on-premise application hosting by utilizing solutions rooted in automation and cloud-based infrastructure. At Polyrific, we have developed an outstanding NoOps solution called Catapult and we offer this article in hopes that it helps you better understand why Catapult is such a big deal.

From DevOps to NoOps

Perhaps the best way to begin understanding the NoOps movement is to first understand the DevOps movement. The term "DevOps" is an amalgamation of "Development" and "Operations" and refers to the interplay between software developers and IT operations during the process of deploying applications to the world. In every enterprise, it is necessary for these two departments to stay close to one another in order to best serve the needs of the business.

At most enterprises, responsibilities for developers generally include the following:

  • Work with stakeholders to understand the needs of the business
  • Distill those needs into requirements and specifications
  • Develop applications that fulfill said requirements

By contrast, IT operations are generally responsible for interfacing with network hardware:

  • Allocation & management of server resources
  • Fault planning & monitoring
  • Security & compliance
  • Device management

Obviously, applications that are developed to suit the needs of the business have to be deployed somewhere so that they can be consumed, and this is where the interplay between the developers and IT operations managers comes in: they must work together to take the developer's work and deploy it to the world on their enterprises resources. This makes perfect sense if the picture were so simple but, as we will see in the next section, reality is a bit more complicated.

Agile & Continuous Deployment

In the early days of enterprise software solutions, very few enterprises created custom software solutions or applications of their own. However, as workplace environments have become more dynamic and reliant on smart hardware and software solutions, the demand for rapid release of custom software applications has grown dramatically. The Agile movement was largely a response to this exponential growth in application demand and it is founded on principles inspired by the Silicon Valley "fail fast & fail early" philosophy.  Gone are the days of months of planning, tedious software architecture design, and software release schedules following a waterfall schedule into a deployment phase that is given equal weight by the IT operations team. Today's software development teams are expected to respond immediately to a seemingly never-ending stream of features and demands requested by the business.

Often, projects are started as bare-bones applications that are immediately thrust into production environments where they will be constantly updated and expanded upon as the business requirements evolve. This sounds great, but it presents a few challenges to software development and IT operations teams, especially with regards to quality of the end-user experience and application uptime. To counter this, the development and ops teams employ a set of automation tools and checkpoints, collectively referred to as "Continuous Integration" or "Continuous Deployment" that smooth out the problems caused by rapid iterations in the software development life cycle. For example, when properly configured, and CI pipeline can trigger a series of automated tests whenever a developer checks in new code to ensure that the new code does not break anything or cause "regression" bugs.

The (Traditional) IT Bottleneck

It operations experts are fantastic but, in our view, their role is best executed when the evidence of their work is everywhere, but their presence is not so apparent. A good server at a restaurant will keep your glass full and your food coming without you noticing them much at all and it should be the same with IT operations managers but sometimes--often through no fault of their own--this is not the case. Without considerable depth of automation in your software development life cycle (SDLC), it becomes necessary for the development team to spend significantly more time with the IT operations team in order to coordinate downtime, deployments, rollbacks, and so forth. This is especially true in the case of on-premise deployments. This close coupling between IT ops folk and the developers is bad for at least three reasons:

  1. It takes the developer's focus away from understanding the needs of the business stakeholders
  2. It cuts into development time
  3. It can influence the engineering and delivery schedule of the application

Given the above, you can probably start to see where this is headed: interaction between development and IT operations should be automated so that the software engineers can remain focused on what they do best: delivering application-based solutions that serve the immediate needs of the business.

NoOps Produces Better Outcomes

So in order to respond to ever-changing demands of the business, development teams must be capable of quickly organizing the stakeholder's needs into business requirements and then parlaying those requirements into working code that is tested, quality assured, accepted by the end user and deployed into production environment on a frequent and recurring basis without being slowed down or distracted by hardware and deployment challenges on the IT ops side of things. Does this mean that IT operations professionals must be removed from the SDLC? Of course not. What it does mean is that IT operations personnel should join forces with the developers to implement game-changing solutions that help to automate the business of getting the developer's changes into production with very little interfacing required between development and operations.

In a NoOps world, developers don't check with IT operations before deploying code or to schedule downtime. In fact, they don't deploy code at all--they simply check their changes into source control and the rest happens automatically, behind-the-scenes, just like the server who always keeps your drink full without your noticing they were there at all. Similarly, developers do not need to request allocation of new resources from the IT department. They can, in theory, "spin up" a new ecosystem of server and database environments for a special purpose app while they sit with the stakeholder during a requirements gathering session.

The Catapult Digital Developer & NoOps Solution

As previously mentioned, we have developed a software solution called Catapult that takes automation of enterprise software delivery to the extreme. Using Catapult, even non-technical stakeholders can create new application projects on a meta-level that immediately spin up server resources using popular cloud platforms such as Azure and AWS. Catapult then allows entry of high-level data models in order to populate databases (or it can connect to existing ones) and generate, and deploy comprehensive codebases all without the user knowing how to write the simplest of SQL queries.

Like the restaurant server that deftly keeps your needs satisfied without making his or her presence known, Catapult allocates hardware resources, creates code bases, sets up source control repositories, allows stakeholders to manage content and seed test data, manages branching strategies, communicates with engineering team members to let them know of code changes, and pretty much anything else a competent developer and IT operations professional on your team would do. That is why we refer to Catapult as the "enterprise digital developer".

If you'd like to learn more about Catapult or any of our other software development solutions, please contact us or call us at 833-POLYRIFIC.

Team Polyrific | Apr 13, 2017

X-Ray vision. Telepathy. Telekinesis. These are the powers that have captured many childhood--and adult--imaginations since the first Superman comic book hit store shelves in 1938. Who hasn't dreamed of being a superhero? Thanks to advancements in wearable technology, we might all have our chance. 

Back in 2015 wearable technologies, or "wearables", hit the consumer market in a full-on assault led by tech giants such as Google, Apple, and Samsung. These tech giants bet big on fast consumer adoption resulting in record-breaking profits. . .and they lost. Then, something interesting happened that is rarely seen in the world of tech: enterprises began adopting technologies originally intended for consumers on a massive scale thereby keeping the market for the erstwhile "game-changing technology" alive.

DHL began incorporating smart glasses in the warehouse to speed up the process of picking orders, Quebec City International Airport equipped their duty managers with Apple Watches enabling them to receive real-time operational alerts with a quick glance at their wrist so that they can make better decisions and decrease delays, Buffalo Wings & Rings restaurant put the new Samsung Gear S3 to work notifying servers when customers need attention without requiring the use of their hands and shaving critical minutes off of the table-to-check cycle which boosts turnover and revenue.

The thing is, wearables solve a problem more critical to the enterprise than to the individual consumer: it multiplies the capability and productivity of a worker while keeping his or her hands free. In essence, wearable technology gives works an extra set of hands. In the case of Buffalo Wings & Rings this means that servers are notified the moment that a new table is seated and immediately when service is needed. This translates to faster turn-around time on tables and, when boosting sales each night by 10-20% is the result, it's pretty significant.

The use of wearables in the enterprise uncovers the potential to take leaps in productivity the likes of which have not been seen since the industrial revolution. Imagine equipping construction workers with smart glasses that allow them to see inside walls or underground so they can locate existing utilities or using smartwatches on knowledge workers to detect when they are at their desk (and therefore available for calls) or when they are becoming stressed or losing the ability to pay attention in a meeting which has carried on for too long.

Wearable technology is going to be the most popular trend in the enterprises over the next couple of years with sales expected to reach $53.2 billion by 2019. This technology has many different workplace benefits and plenty of options available to suit the individual use-cases of almost any enterprise. In addition to smart watches and glasses, smart clothing is now emerging. In March 2017, Levi's and Google announced a partnership to develop a smart jacket that allows it's owner to interact via gestures such as brushing a hand on a sleeve. This may sound silly at first glance, but imagine a field worker, soldier, fire-fighter, or any other type of worker who is wearing bulky clothing and possibly gloves that make interacting with a tablet difficult--to them it's not so silly of an idea.

Wearables also can collect myriad biometrics from their users which may or may not be subject of privacy debate in your enterprise. Assuming that your employees are willing to grant access to their personal biometric data, there are many interesting insights that may come from it such as levels of stress and fatigue, including stress when in proximity to another specific worker.

Wearable technology indeed has the ability to grant superpowers to workers of all types in your enterprise. Your only limitation is your imagination (and ours if you hire us) but one thing is for certain: you will need a great technology consultant and application developer to make your vision become a reality and we want to be the consultant that helps you become a superhero in your enterprise.

PRO TIP:  Begin integrating wearable technology into your enterprise by identifying tasks that require employees to use their hands while at the same time requiring them to refer to various data sources. Whenever you find a situation like this, there is almost certainly a solution that can be provided by the use of wearable technology.

Please contact us today learn more about wearable technology and how it can be used at your enterprise.