Development of Methodology for Performance Assessment and the Effectiveness of Innovation Activity

The article presents the results of the research that is being carried out with the financial assistance of the Russian Federation Government represented by the Ministry of Education and Science of the Russian Federation. The unique identifier reference number for the project is RFMEFI60115X0007. The hypothesis of the paper supposes that innovation process can be decomposed to the level of separate stages, whose efficiency and effectiveness can be assessed with the help of suitable key performance indicators (KPI).


Introduction 1.
Economy innovativeness greatly influences the development of society and the country as a whole.The World Economic Forum (WEF) publishes a report on the competitiveness of different countries annually.The WEF regards innovations as one of the key factors that determine the competitiveness of a given country.It is possible to classify the three stages of country development on the basis of assessing the factors relating to their competitiveness: 1) resource development; 2) efficiency growth; 3) innovative development.In the 2011 report by the WEF the development stage of the Russian economy was identified as the efficiency growth stage (The GlobalCompetitiveness..., 2011).At the same time, the economy of China, one of the countries along with Brazil, India and South Africa (BRICS) which the development of the Russian economy is often compared to, is in the transition from the second to the third stage due to their ongoing development of innovations.Brazil by comparison is on the second stage while India is on the first stage.Effectiveness improvements of the Russian economy through better education, the growth of markets efficiency and a rise in labour productivity, the development of financial markets, techniques and transparency will allow for a shift to the third development stage, the framework and key priorities of which will be given to the development levels of companies' business-models and the use of innovations (TheGlobalCompetitiveness..., 2011).
Despite the importance of innovations in assessing the competitiveness of separate organizations and economies as a whole, there is no clear understanding of innovation activity nowadays.From an economic point of view, the key task of innovations is to create additional value for consumers as well as for the companies implementing said innovations.Companies conduct different types of innovation activity including the development of new goods and services, an increase in labour productivity, the improvement of organizational and marketing processes, the improvement of quality and availability of their goods and services, etc.However, currently, the focus of research into the innovation activity of companies has shifted to the development of new technologies and solutions, while at the same time the integrated nature of innovations is ignored.It is absolutely crucial to overcome such a technologically focused approach to innovative company performance.
Academic sources do not have unanimous opinion on how to assess, measure and introduce innovations, taking into account the fact that there are different models of managing companies and of corporate innovative systems in education, industry, agriculture and medicine.

2.
The basic interpretation of innovation activity in international practice is the highlighted in the OSLO Manual.According to this manual innovation activity includes all scientific, technological, organizational, financial and commercial activities that really lead to the implementation of innovations or that are designed for that purpose.
The definition of innovation activity in Russian legislation is similar to the above mentioned.Federal Law No. 127-FZ provides the following basic definition: innovation activity is the activity (including scientific, technological, organizational, financial and commercial activity) directed at the implementation of innovation projects and at the creation of innovation infrastructure and the provision of its activity.
Both definitions provide the common feature of any innovation -that it should be brought into use, i.e. implemented.A new or improved product is considered to be brought into use when it enters the market.
Thus, the innovation process of the upper level consists of three main phases: 1. Invention 2. Acquisition 3. Usage.Invention implies idea generation, research and technological development and elaboration to create a prototype; usage implies the process of internal or external commercialization and capitalization.Figure 1 presents the whole process in more detail.

Figure 1. A typical process of innovation activity
According to the typical process, innovation passes through four main stages which can vary depending on the specific nature of the corporate innovative platform (system) built in the organization and on its performance features.
The general viability of innovation activity beginning is determined by the continued possibilities of expanding its share on the prospective market or to acquire new internal possibilities at the expense of innovations.
• Stage 1: Generation and Selection of Ideas -consisting of two stages of ideas management where the idea is generated, assessed and selected by a corporate innovations business unit who will carry out further research and development.• Stage 2: Research and Development -the transformation of the idea into a developed and financially viable innovation decision.As a result, ideas can turn into a prototype of an intangible asset of the organization.The alternative for each stage is the acquisition of formed ideas or off-the-shelf solutions on the market.The choice between independent development or outside purchase is made based on a number of factors including cost relationship, organization's employees possessing necessary competencies, production risk, efficiency of different options, etc. • Stage 3: Commercialization of Innovations -this can be fulfilled internally or externally.External commercialization implies launching an innovation into the market with further promotion and distribution.
Internal commercialization is implemented if the company plans to use the developed innovation for its own use, without reselling it to external consumers.• Stage 4: Innovation realization -increments of the financial/non-financial benefits of the organization at the expense of implemented innovations.An innovation process can be considered successfully completed when it has acquired the following improvements types (according to the OSLO Manual): 1. Product innovation -the introduction of a good(s) or a service that is new or significantly improved in its quality or utilization.This innovation includes significant improvements in technical standards, components and materials, in embedded software and in the degree of user friendliness or other functional characteristics.
Process innovation -the introduction of a new or significantly improved production method or product delivery method.This innovation includes significant changes in technology, capital equipment and/or software.Marketing innovation -the introduction of a new marketing method, including significant changes in the design and packaging of the product, its storage, market promotion or by setting a market price.2. Organizational innovation -the introduction of a new organizational method within the company's business practice, in workplace engineering or external relations.However, the absence of such introductions does not mean the failure of an innovation process as the failures in this sphere occur much more often than successes and are attributed to a variety of factors.But the very experience gained during the whole process can be really valuable and of use further down the line.That is why any and all assessments of effectiveness should include these aspects.
Different information sources including open data on innovationally-developed organizations, research by consulting firms, literary sources (articles) and interviews with experts and managers in the field are used to form the list of indicators used to assess innovation activity and the conditions of success of corporate innovation systems.
1.The rating of 50 of the most innovative companies from the Boston Consulting Group forms the basis to carry out benchmarking.This rating has been made annually since 2005 classifying industries by their dynamics in the following indexes (in % to the previous year): -Revenue; -EBIT -Earnings Before Interest and Taxes; -TSR -Total Shareholder Return; -R&D Spending.A standardized pattern to systematize the results of index benchmarking and the conditions of success for every company was developed.
Description of items to carry out analysis is the rating of 50 most innovative companies in 2014.
2. The American Productivity & Quality Center (APQC) is a non-profit organization established in the USA in 1977; it specializes in the field of researching the efficiency of business processes, carrying out comparative analyses of the practices of different companies and identifying best practices.While conducting its research the APQC focuses on the data provided by its member-organizations. Currently there are more than 320 global companies which are members of APQC. 3. Research by consulting firms.As it was mentioned before, not all managers are satisfied with the applied ISSN 2039-9340 (print) Mediterranean Journal of Social Sciences MCSER Publishing, Rome-Italy Vol 6 No 5 S4 October 2015 192 systems of innovation activity assessment.For many of them innovations are 'a black box', which in layman's terms means that they are difficult to manage.That's why the demand for consulting services is rather high and consultants often share their best practices.4. Interviews with experts and managers are a source that allows many issues to be solved and catches nuances that are not evident while gathering data from open sources. 5. Scientific literature, selected according to the following criteria: -Relevance; -Reliability of the sources (the scientific validation of the paper must be proved); -Review of published materials.The composition of the information data base is mostly based on practical sources.It is explained by the fact that the assessment of efficiency and effectiveness is an applied tool and its development requires maximum mundanity to the real activity of organizations-innovators.
The innovation activity of an organization is a complex phenomenon that is difficult to measure with the help of only proxy variables (such as the number of patents or the amount of goods), so the survey technique of Russian managers was chosen as a method to fulfill that task.A coherent approach, to assess manageability and innovation activity, was used to satisfy the aim of the research; the approach involved 15-20 interviews with the managers of innovationally-active Russian companies and then, alongside the suggested hypotheses, were empirically tested on the basis of selection for innovationally-active organizations.
The first, qualitative stage of research implied conducting in-depth interviews with the representatives of organizations from pioneering industries.The second, quantitative stage of research implied holding an opinion poll of the representatives from the selected organizations with the help of a questionnaire.The questionnaire was developed in strict conformity to standard procedures (Churchill, 1979) and Russian statutory provisions.The data base of the Russian company SPARK was used to form the general totality of selected organizations and further verification of their financial data.The quantitative stage of research was based on the interviews with the representatives of innovationally-active Russian organizations.It is assumed that the managers of said innovationally-active Russian organizations from Marketing Departments, R&D Departments or top management will be the respondents.Respondents are chosen relative to their potential awareness about the innovative development of their organizations.
During the qualitative stage of research 15 interviews with the managers of innovationally-active organizations were conducted in the small rule to beta test the chosen aspects of innovation activity for the company.Such choice of the companies helped not only to test the model, but also to detect the specific character of the innovative development of Russian organizations.
The interviews were analyzed with the help of content-analysis which was conducted using such a procedure that allowed a systematic study of the semantic information to be carried out (Krippendorff, 2004).Such an approach considers the mention frequencies of terms and phrases that are logically connected with the factors of the analytical model.Thus, using only qualitative indicators one could identify the availability of the model components and the evidence of a cause and effects relationship between them.Such characteristics of content-analysis as emic validity, coefficient of coders agreement and the mention frequency of separate model components were identified after decoding, coding and reducing data.
The next stage dealt with the calculation of the indicator for the touch points of manageability and innovation activity (for such components as a new product, technological base, clients' decisions, etc.) by adding mentions of individual points from the coding form.Then, the nonparametric Spearman correlation coefficient (ps) to rank data was calculated.To recognize the relationship between model components, the significant correlation coefficient should exceed the threshold value 0.50.
During the quantitative stage of research and to conduct a complex analysis of the innovation activity of organizations it was necessary to choose industries in which innovationally-active organizations were mostly represented.The sampling of quantitative research included companies from 5 Russian regions: Moscow and the Moscow region, Saint-Petersburg, Rostov-on-Don and Tomsk.
Variables description: Questions within the questionnaire measuring the three aspects of innovation activity of the organization, and indicators of its manageability and efficiency were formulated for the quantitative stage of research.The main variables that are used in the research are operationalized on the basis of published materials from previous studies and on the results of the qualitative stage of research.Aspects of innovations activity of the organization, i.e. the development of new goods, technological innovations and external innovations (on the example of innovations in distribution channels), are based on the methodology developed by us.The questions were formulated in the following way: "How far do you agree with the following statements?"; a 7-point Likert scale was used to answer the questions (1-'absolutely disagree', 4-'I don't know whether I agree or disagree.7-' absolutely agree').
The 'New Product' factor was measured with the help of the scale that consisted of four questions which could identify the ability of the organization to create new products and to introduce them into the market and be competitive on their basis.To assess the 'Technical Base' factor there were questions which assessed the ability of the organization to use technologies in order to attract consumers, and to introduce new goods and services and to reduce costs.The 'Distribution Channel' factor was measured with the help of a set of four questions that characterized the ability of the organization to introduce innovations into distribution channels, to adapt them to new customer segments and to influence intermediaries.
The measurement of the key resulting variables of the organization's activity was based upon three groups of questions.Firstly, the manageability of organizations is measured according to answers to the questions about the ability to configure and to adapt business-processes, the ability to communicate efficiently with partners and the ability to achieve goals cooperatively.Secondly, the influence of manageability on the innovation activity of organizations was measured (Fig. 2).Thirdly, value to customer was identified with the help of organization's customers satisfaction score, customer loyalty and the significance of the good/service for customers.The measurement procedure was based on methodology offered by (Venkatraman, 1989) and modified further by (Vorhies and Harker, 2000).
The test variables were set as the managers' assessment of industry growth (MKT_GRi -annual average increment for the last three years in per cent) and the competitive intensity in the market (COMPi) measured as a joint market share of the three biggest competitors within the large market of the given company.
Description of data analysis procedure: While analyzing we used the standard methods of psychometric research that are applied when questioning (Hatcher, 1994;Hu and Bentler, 1999).Firstly, the reliability coefficient, (Cronbach's Alpha) for all variables measured on the basis of scale consisting of several questions, was assessed.Calculated coefficients should correspond to the recommended minimum reliability level of 0.75.Then, a factor analysis was conducted using the principal components method for 12 questions describing aspects of innovation activity.According to recommendations on conducting factor analysis using the principal components method (Hatcher, 1994) one should use the principal axis method and orthogonal rotation that maximizes variation (varimax).
Conducted analysis proves the identified aspects of innovation activity that are measured with the help of factors or latent variables.Received factor score is used in further regression analysis.Thus, the value of the individual aspects of innovation activity for a separate organization equal the value of factors received during factor analysis.
Likewise, factor analysis using the principal components method for manageability and efficiency of the organization (growth, earning power, value engineering for customers) was conducted; factor analysis also consisted of 12 questions.
So, the results of factor analysis are used to calculate the indexes of innovation activity and the efficiency of organizations employed in regression analysis in the following model (1): Y1 -the index of a company's efficiency PROD -creation of innovation products TECH -technological innovations TSEN -creation of innovation value to customers COMP -competition level in the field (which can include other factors as well -from survey) -level of industry growth (which is absolutely necessary as it shows the growth of organizations regardless of the field).
(1) Standardized and non-standardized regression coefficients are identified with the help of the maximum likelihood method.Non-standardized coefficients are used to test hypotheses; standardized coefficients are used to determine factors that mostly influence the efficiency and innovation activity of the organization.

3.
An analysis of the executives' answers received during questioning allowed for the formation of a list of guiding principles, following which it is possible to design an effective measurement system for the innovation activity of an organization.
• Multidimensionality.The system must include a combination of financial and non-financial indicators to assess the various aspects of activity.It should also consider the possibility of adjustment for lagging and advance indicators to analyze current activity and activity during the past and targeted periods.• Process approach.Using the process approach in the formula 'Innovation=Invention+Usage' permits a description of the innovation activity of a company as a business process that consists of four main stages: generation and selection of ideas; research and development; commercialization; fulfillment of innovation.• Aiming at reaching stakeholders' goals.The system of innovation activity assessment must be efficiently built in accordance with an organization's goals and the goals of a wide range of stakeholders -shareholders, customers, employees, government watchdogs and society as a whole.• Cause-and-effect relationship.Assessment indexes in the system should be logically connected on the basis of mathematical calculations or cause-and-effect relationship.Such links will allow organizations to identify the results of various approaches and whether it is worth starting at all.• Ease of introduction and usage.An assessment system should be simple and understandable enough so that organizations can incorporate it into their internal systems.The variety of modern organizational structures requires system flexibility to cascade assessment indicators into the hierarchical levels of the organization.To match the given principles, the assessment system should be incorporated smoothly into the corporate innovation system of an organization.We suggest the use of the 'Innovation Spiral' model by the Ernst & Young Company.This model regards the corporate innovation system as a structured cyclical process of the upper level which includes the following sections (see Fig. 3): 1. External environment -market and economic conditions which influence the company; 2. Business outcomes -profit growth, business sustainability, increase in labour efficiency, etc.; 3. Innovation enabler -the internal conditions of an organization that ensure a positive output of the 'Innovation Spiral'; 4. Interaction with stakeholders -external and internal collaboration to gain knowledge, to attract competencies and to consider opinions; 5. Innovation process -direct activity that leads to innovations; 6. Areas of innovation that the company is engaged in; 7. Competitive advantage -the final outcome of the 'Innovation Spiral'; according to J. Schumpeter the innovation concept implies getting a competitive advantage at the expense of a temporary monopoly acquired due to the introduced innovation.

Figure 3. Innovation Spiral
So, this model of a corporate innovation system meets the five guiding principles of developing the assessment system of efficiency and effectiveness for innovative activity of an organization defined earlier.
An important benefit of the 'spiral' is its 'free layout', that is, the possibility of filling the sections with necessary elements depending on the targets.
To keep the assessment system for the efficiency and effectiveness of innovative activity simple, easy to introduce and use, it should include the following elements (see Table 1.): 1. Description of typical stages of the innovation process; 2. Recommendations on how to form success conditions in an organization; 3. A model for performance assessment -a work mask that identifies and groups financial and non-financial key performance indicators (KPI) of each stage in the optimal way; 4. Target values and benchmarks for groups of KPI; 5. Reporting forms based on the assessment results; 6. Methods and infrastructure for capturing, analyzing and interpreting data, and their dissemination in reporting forms.
Table 1.Assessment system for the efficiency and effectiveness of innovation activity.

Discussion and Conclusion 4.
The measurement of innovation activity and efficiency is not an easy task even within a single organization as innovation ISSN 2039-9340 (print) Mediterranean Journal of Social Sciences MCSER Publishing, Rome-Italy Vol 6 No 5 S4 October 2015 196 processes are greatly influenced by a number of external and internal factors and also because there are various approaches.Academic sources do not provide some unanimous opinion on how to define, measure and introduce innovations.Moreover, one can identify three main restrictions in innovation research.Firstly, the term 'innovation' identified is rather narrow and specific: innovations often imply only the development of new products or changes to products or organizational processes (Howells et al., 2006;Sawhney et al., 2006;Van de Ven, 1986).Secondly, there is no uniform classification for innovations.Many researchers apply a dichotomous system of innovation division into, for example, radical (breakthrough) and incremental (step-by-step) innovations (Tushman, Anderson, 1986).Such a division simplifies a complex phenomenon of innovations up to one or two features on the basis of which the classification is developed.Thirdly, this restriction is connected with the tendency to measure innovations with the help of simplified and aggregated secondary indicators such as the number of patents or the size of budget for the development of new products.
As a result of a number of sectoral studies, dissatisfaction within companies with the existing assessment systems of innovation activity was discovered.Thus, according to the research of the Boston Consulting Group (BCG), less than 41% of managers from 1600 innovation organizations in the world consider their assessment system effective, though the majority needs to improve the system.According to the received data, organizations interested in the improvement of the results of innovation activity should review the assessment system that they use.Many systems are described in scientific sources, but from a practical point of view, none of them meets the requirements of innovationally-active organizations.
From our point of view, the 'innovation enabler', 'innovation process' and a relevant set of KPI are key sectors of the 'Innovation Spiral' required to develop the assessment methodology.In the frame of our paper, the given sectors are considered in details and recommendations for their completion with specific elements are formulated.

Figure 2 .
Figure 2. Influence of innovation activity on the manageability and efficiency of organizations.