BIM Show Live 2013_Slides.pdf · In studies, the top 25% tend to think that their skills are . in...

Rory VanceKnowledgeSmart

Email: rory.vance@knowledgesmart.netWeb: http://www.knowledgesmart.netBlog: http://the-knowledgesmart-blog.blogspot.comTwitter: http://twitter.com/knowledgesmart

BIM Show LiveLondon30 April – 01 May 2013

Self-Assessment

How effective is it?

Statistically speaking...

63% of Americans consider themselves more intelligent than the average American.

Whilst 70% of Canadians said they considered themselves smarter than the average Canadian.

In a survey of engineers 42% thought their work ranked in the top 5% among their peers.

A survey of college professors revealed that 94% thought they do ‘‘above average’’ work – a figure that defies mathematical plausibility!

Imperfect Self-Assessment

The “above-average effect” is the tendency of the average person to believe he or she is above average, a result that defies the logic of statistics.

In studies, participants scoring in the bottom quartile on tests grossly overestimate their performance and ability.

Although test scores put them in the 12th percentile they estimate themselves to be in the 62nd.

Conversely...

Because top performers find the tests they confront to beeasy, they mistakenly assume that their peers find the tests to be equally easy. As such, their own performances seem unexceptional.

In studies, the top 25% tend to think that their skills are in the 70th–75th percentile, although their performances fall roughly in the 87th percentile.

Conclusions

For a given skill, less capable people will:

tend to overestimate their own level of skill;

fail to recognize genuine skill in others;

fail to recognize the extremity of their inadequacy;

recognize and acknowledge their own previous lack of skill, if they can be trained to substantially improve.

The Knowledge ‘X-Factor’

“In essence, our incompetence masks our ability to recognize our incompetence”.

Measuring Performance

3 Levels of KnowledgeExample: Revit Architecture

Not all users need to become Revit 'experts'. Many people have a primary role which means they come into contact with Revit on an occasional basis, but no more than that. For example, PM's, Project Architects, Practice Principals, and so on.

Our ‘Level 1' assessment, called 'Revit Architecture for occasional users’ looks at some basic concepts, including: Files & File Formats, Navigation, Views & Sheets, Measuring, Exporting Data, Families and Element Selection.

Measuring Performance

For the primary modelling team, we have 'Level 2', or 'Revit fundamentals' level material. This is a general level test, covering a wider range of topics, including: Basic Element Creation, Views & Sheets, Detailing, Keynoting & Annotation, Worksharing, Dimensions & Rules, Interoperability, Families & Parts, Scheduling, Coordinates & Orientation and Outputs.

This level of test material can be used to create reliable benchmark data for the firm, compared to industry average statistics. Performance 'quartiles‘ are an effective way of targeting users with incremental productivity improvements over time.

Measuring Performance

Performance Quartiles

Measuring Performance

Advanced Users

For users placing in upper quartile 3 or quartile 4, we have a variety of modules which are designed to address more advanced Revit concepts.

The 'Level 3‘ material covers more process based scenarios and looks at the impact of using the software in a project environment. Topics such as Revit Families, Work flows, Project process and Worksharingare covered in greater detail.

Learning Plateaus

A learning plateau is when forward progress seems to have stopped while engaged in learning a new skill.

http://the-knowledgesmart-blog.blogspot.co.uk/2013/03/learning-plateaus.html

AEC Metrics

AutoCAD vs MicroStation59% in 78 mins 58% in 82 mins

0-1 yrs vs 4-5 yrs vs 8-10 yrs

53% in 87 mins 60% in 77 mins 61% in 79 mins

User data capture

A key goal looking ahead is to capture and share more detailed user demographics.

- For how many years have you been using BIM/CAD/Engineering software?

- How often do you use BIM/CAD/Engineering software?- What BIM/CAD/Engineering software do you regularly use? - How did you primarily learn to use BIM/CAD/Engineering software? - Where did you first learn to use BIM/CAD/Engineering software?

Anything else?

Self-rating software skills

Please rate yourself on your ability to use the software, for which you are about to take a test, on a scale of 1 to 5:

1 = Very basic knowledge; not enough to work confidently on a project

2 = Basic knowledge; can get by working on a project but could do better

3 = Good knowledge; can produce a good standard of work on projects

4 = Advanced knowledge; can teach the basics to others

5 = Expert knowledge; can perform and teach others at an advanced level

Dreyfus model of skill acquisition

A model of how students acquire skills through formal instruction. Brothers Stuart Dreyfus and Hubert Dreyfus proposed the model in 1980 in a report on their research at the UCAL Operations Research Center for the USAF Office of Scientific Research.

The model proposes that a student passes through 5 stages:

Novice Advanced beginner Competent Proficient Expert

Skills Gaps

8-step guide to effective Training Needs Analysis

Step 1 – searching your results

Search your results across a range of categories, for example; min score, max score, elapsed time, user name, test name & ID, date ranges, training meta tags, plus 5 extra datafields.

Step 2 – grouping your results

Create new sub-groups of your data, using the grouping tools.

For example, all users who scored 60% or less on basic AutoCAD; all users who took longer than 90 mins to complete a basic MicroStation test; all users who require training on Worksets in Revit Architecture; and so on.

Step 3 – charting

This chart gives you an easy at-a-glance way of identifying the most urgent training topics, for a given group of users.

This chart helps you to monitor question scores for each test in your library and further recognise any potential problem areas.

Step 4 – group comparison

Use the Group Comparisoncharts to analyze and compare results across a range of search results.

The Global Comparison chart offers you an external benchmark, for our most popular test titles.

Step 5 – personal training curriculum

Export your results data to csv.

When you export results, you will see a field called, 'Training Tags’; a personal summary of training topics for each user.

Step 6 – review results

Schedule a one-to-one with your users, to discuss theirresults and training recommendations summary.

Step 7 – links to learning

Follow up with modular training workshops, which specifically target the highlighted skills gaps.

Group people together according to experience and ability. For example, avoid training novice users and expert users in the same session.

Use the KS meta tags to link to corresponding modular learning content from third parties (i.e. Evolve Consultancy, White Frog, 4D, Global e-Training, ASCENT, Bentley Learn, ATC's, etc.) and your own in-house training collateral.

Step 8 – sharing knowledge

Identify your best users; they will appear in Quartile 1 or Quartile 2 on your Performance Spread Charts.

Challenge your leaders to help create modular training material, which you can then use to share knowledge and best practice across the firm.

Tag your learning material, so it can be searched and easily found via your firm's intranet, LMS, or similar.

Thank You!

Email: rory.vance@knowledgesmart.netWeb: http://www.knowledgesmart.netBlog: http://the-knowledgesmart-blog.blogspot.comTwitter: http://twitter.com/knowledgesmart

BIM Show LiveLondon30 April – 01 May 2013