How to play with statistics …

Soon after this letter was sent to Oak Bay News, District of Oak Bay Planning Department, the Mayor and Councillors, the OBUC posted their analysis of the “push polls” that they called public opinion surveys.
Below (and attached) is a detailed response from a retired town planner who is very experienced in understanding these matters.

OBUC Public Opinion Surveys Report – July 25, 2018

Open House Surveys
On 25th, 28th and 29th April 2018, three public Open Houses of four hours each were attended by 222 people. 182 surveys were completed covering subjects of interest in determining attendees' opinion about Oak Bay United Church's proposed affordable housing development project. The survey questions were drafted by big think with analysis of the results by big think and Pivotal. Attendance was tracked by attendee's stated home location and ≈76% were from Oak Bay, with several others from just over the Victoria boundary.

Definition of near neighbour:
Note that postal codes are defined by individual "walks," so the definition used as "neighbour" although intended as an approximate two block radius, was shortened to take account of Oak Bay's municipal boundary – which runs along Foul Bay Road – but lengthened in other areas e.g. east almost to Monterey and south almost to McNeill) to more reasonably cover input. NOTE: It is questionable whether this fairly wide area really is the area of ‘near neighbours’. As a result, the reported results may not accurately reflect ‘near neighbour’ views.
Note: the transmittal letter signed by Chris Corps CEO, Pivotal IRM Inc. – he is also the OBUC Development Project Manager.

 Survey Questions and Responses

 #1 – Support affordable housing in Oak Bay
Do you think Oak Bay would benefit from affordable housing for seniors, young professionals, young families, single parents etc.? Answers: Yes; Somewhat; No.

This question was framed to explore overall support for affordable housing, irrespective of the plans and proposal. ·         # of responses - Overall 176  – Non-OB 43, All Oak Bay 133, Neighbours 55, Rest of Oak Bay 78).

Table 1

Table 1.jpg

·         As reported with ‘yes and somewhat’ answers combined: Overall support = 96%, Non Oak Bay = 100%, All Oak Bay = 95%, Neighbours = 93%, Rest of Oak Bay = 96%.
·         As reported, with ‘yes, somewhat and no’ separated and non- Oak Bay residents excluded, the numbers and percentages indicating support change considerably:

 Table 2

Table 2.jpg

 #2 – Design Appeal

Overall, is the proposed housing design appealing? Answers: Yes; Somewhat; No.
This question was framed to draw out whether there is general support for the design.

·         # of responses - Overall 179  – Non-OB 43, All Oak Bay 136, Neighbours 56, Rest of Oak Bay 80.

Table 3

Table 3.jpg

·         As reported, with ‘yes and somewhat’ answers combined: Overall support = 69%, Non Oak Bay = 89%, All Oak Bay = 63%, Neighbours = 50%, Rest of Oak Bay = 72%.
·         As reported, with ‘yes, somewhat and no’ separated and non- Oak Bay residents excluded, the numbers and percentages indicating support change considerably:

Table 4

Table 4.jpg

·         Yet the report states: Overall more than two thirds favour the design and there is also overall support for the design within Oak Bay (by almost 2:1). The most sensitive group – immediate neighbours – were evenly split over the design. This is not accurate.
Note: As the near neighbours have said all along, they are not opposed to the provision of affordable housing – and the open house surveys demonstrate this.

 #3 – Multi-Family Suitability

The shape, form and size of proposed building is consistent with nearby multi-unit neighbourhood buildings. Answers: Strongly agree; Agree; Neutral; Disagree; Strongly disagree
·         # of responses – Overall 176 (97%), Non-OB 42 (23%), All Oak Bay 134 (74%), Neighbours 57 (31%), Rest of Oak Bay 77 (44%).

Table 5

Table 5.jpg

·         As reported, with ‘strongly agree, agree and neutral’ answers combined: All responses = 72%, Non Oak Bay = 91%, All Oak Bay = 65%, Neighbours = 45%, Rest of Oak Bay = 80%.
·         As reported, with ‘yes, somewhat and no’ separated and non- Oak Bay residents excluded, the numbers and percentages indicating support change considerably.

Table 6

Table 6.jpg

The report states “This question was framed to judge acceptance as a multi-family design in the context of surrounding multi-family buildings. The report goes on to comment: “This question was intended to test neutrality as there are almost no similar multi-family designs nearby, so responses should in theory have noted low consistency with multifamily buildings. Respondents therefore were most likely responding to aspects not included in the question.
Note: respondents could hardly have missed Granite House immediately across the street from the Open House, so the above comments make no sense.

 #4 – Traffic and Parking
The design adequately addresses parking and congestion considerations. Answers: Strongly agree; Agree; Neutral; Disagree; Strongly disagree
This question was framed to see if attendees felt traffic and parking considerations had been adequately addressed, as this had been a concern expressed at the Neighbourhood Sessions.
·         # of responses – Overall 175 (97%), Non-OB 42 (23%), All Oak Bay 133 (73%), Neighbours 57 (31%), Rest of Oak Bay 76 (42%).

Table 7

Table 7.jpg

Table 8

Table 8.jpg

The report states “Little direct information was provided about parking ratios or traffic, as studies had not been available, but draft results were known to OBUC advisors. A good number of attendees asked questions however. It is probable that responses reflect general perceptions about traffic and parking, irrespective of information available and answers provided.
The report goes on to comment: that “Traffic and parking are common concerns, not limited to Oak Bay. Questions at the Open Houses on parking ratios seemed to satisfy most people, which is borne out in the survey, albeit with neighbours continuing to be concerned about traffic on Granite Street, almost all of which is unrelated to and will be almost unaffected by development.”
Note: Most respondents, particularly those who do not frequent the area regularly would have been largely unaware of traffic and parking conditions in the area and as, the authors admit, little direct information was provided, underlining why the above comments in the report make little or no sense.

#5 – Overall Suitability

The building design fits in with the overall character of the Oak Bay neighbourhood. Answers: Strongly agree; Agree; Neutral; Disagree; Strongly disagree
This question was framed to assess whether attendees felt the building fit the neighbourhood.
This question was intended to address the heart of the issue, namely whether the design is considered suitable for the site.
·         # of responses – Overall 178 (98%), Non-OB 43 (24%), All Oak Bay 135 (75%), Neighbours 56 (31%), Rest of Oak Bay 79 (44%).

Table 9

Table 9.jpg

Table 10

Table 10.jpg

The report states that general Oak Bay respondents found the design acceptable - 67% of the rest of Oak Bay residents were favourable (i.e. excluding neighbours); NOTE: this includes those who were neutral.  
Note: Nowhere in the survey is there mention that on 3 sides (and part of the 4th side) the property is surrounded by 1-2 storey houses. The only mention of context was to multi-storey buildings. For those who are not familiar with the area, this could skew the results.

The report also comments at some length about the behaviour and attitudes of neighbours.
·         “While a minority of neighbours supported the design for the site, the minority was only a difference of five people, which is small in the context of broader views.”
·         “During the Open House sessions, some identifying themselves as having "stop development" boards on their lawns privately noted they had done this due to pressure but were generally happy with the proposals. This is borne out by survey returns.”
·         “Across most developments, neighbours' opinions are typically the most anti-development, because proposals represent change to their community and it is often difficult for them to imagine how development may contribute positively. Results here are no different. Acceptable density and height were the main comments received. In general neighbours said the development would be acceptable at three storeys, (≈9' difference in storey height).”
Note: The remarks quoted purportedly from neighbours are anecdotal and cannot be verified. People have not been ‘pressured’ to take signs. If people didn’t want one, they were not given one, plain and simple. No where did the survey ask about height – so how many actually indicated that 3 storeys would be acceptable?

 #6 – Privacy

Privacy concerns have been addressed through the planning and design. Answers: Strongly agree; Agree; Neutral; Disagree; Strongly disagree
This question was framed to assess whether enough had been done to mitigate neighbours' privacy in the design. ·         # of responses – Overall 162 (90%), Non-OB 38 (21%), All Oak Bay 124 (69%), Neighbours 53 (29%), Rest of Oak Bay 71 (39%).

Table 11

tABLE 11.jpg

Table 12

Table 12.jpg

 The report states that the “majority of neighbours (57%) felt the steps taken in the design had addressed privacy concerns”, however this includes those who were neutral (30% of responses). The report also contends that “the general consensus (75%) is that the design is addressing privacy issues – again this includes neutral responses and responses from non-Oak Bay residents.
Again, there were comments included about immediately adjacent neighbours, with the conclusion drawn that the results are positive. However, only 27% of ’neighbours’ felt privacy issues had been addressed, with 30% neutral and 44% disagreeing.

 #7 – Information Adequacy

Has this open house offered you a better understanding of OBUC affordable housing project? Answers: Yes, Somewhat, No (although the report mistakenly says Strongly agree; Agree; Neutral; Disagree; Strongly disagree)
This sought to assess whether the Open Houses were informative (i.e. helpful).
·         # of responses – Overall 177 (98%), Non-OB 40 (22%), All Oak Bay 137 (76%), Neighbours 57 (31%), Rest of Oak Bay 80 (44%).

Table 13

Table 13.jpg

Table 14

Table 14.jpg

The report states that “Respondents almost unanimously confirmed the Open House sessions were helpful”, but this includes the ‘somewhat’ responses. When those responses are taken out, the level of support is much lower:  only two-thirds of Oak Bay residents.
The report continues: “This (the level of satisfaction with the open houses) is important because it affirms the reliability that the answers to other questions were made with good knowledge about the proposed development.” This conclusion is a stretch or reasoning, nothing else.

#8 – On-line Support

Would you be willing to sign an online pledge in support of the project? Answers: Yes; Somewhat; No
This sought to assess whether respondents would support the project through an on-line form, thus facilitating anonymity. 91% of attendees answered this question, 70% Oak Bay residents, 30% neighbours:

Table 15

Table 15.jpg

The report comments: “Unfortunately the question was interpreted by some as implying commitment to financial expenditure (i.e. "pledge"). This was not intended but acted as a deterrent. Responses are thus indicative of respondents' views. It shows low interest in participation.” “Seeking clarification, some Open House attendees commented their concern was not just with financial commitment, but also that their names might become public and misused by the pressure group. Although they were assured, most didn't enquire and it is assumed this affected results.”
Note: the results might also show that many attending don’t feel passionate about supporting the project.

 #9 – Council Support

Would you be willing to speak in support of the project to Oak Bay council? Answers: Yes; Somewhat; No
This sought to assess whether respondents would appear at Council to speak in favour of the project. 93% of attendees answered this question, 71% Oak Bay residents, 31% neighbours:
The report concludes that “Only one in four are prepared to attend Council and speak positively about the project, yet – as shown in responses to Question 5 – 63% favour the project.” “This survey shows however that the majority favour the project, including a high portion of near neighbours.”

NOTE: These comments are based on the responses about overall suitability, at no time were respondents specifically and directly asked if they supported the project. The number ‘in support’ includes the responses from all respondents, including some 24% who are non-Oak Bay residents and combines Strongly agree, agree and neutral responses. Again, the conclusion the report draws is a stretch. A ‘high proportion” of near neighbours do not support the project: the wording is misleading, only 33% agreed or strongly agreed with its suitability. As well, just how near a neighbour is, is not clear.
The report continues: “Reluctance to appear before Council is a common issue facing developments. People generally have little time or interest in community participation unless it directly affects them or they have a strong opinion. The implication for Council is a responsibility to consider not just the wishes of those that appear before Council, but those that do not take the time to make their views known.”

 Concluding Comments about the Survey Analysis

·         The report’s conclusions are not presented in an un-biased fashion. They are manipulated to show more support for the project, which on careful analysis in many cases is not accurate. The report makes all kinds of claims of support that are not adequately substantiated. They have included ‘somewhat’ and ‘neutral’ comments in the “support” category, although it can well be argued, they could also logically be included in the ‘non-support’ category.
·         Many of the survey questions are open to interpretation, making the results and conclusions drawn questionable. 
·         The report includes non-Oak Bay residents (22% of respondents), in many of the support totals.  
·         The report states “There was a reasonably equal split between those within the immediate neighbourhood (35%) and those from the rest of Oak Bay (43%). This is hardly an equal split. Nowhere does the report indicate where congregation members live.   
·         The lowest percentage of returned surveys was from neighbours (≈73%). The report claims that comments (presumably oral) indicated a reluctance due to perceived "peer pressure" from other neighbours. There are other explanations for a low return by neighbours, e.g. not wanting to participate in an exercise completely controlled by a body that has a vested interest in the results (a question of trust). 
·         The report makes frequent, unflattering (anecdotal and hearsay) comments about the near neighbours. This is unprofessional.
·         The report contends that the open house sessions “included significant influence from an outside pressure group”. This was in reality a very small group – by far and away there were more residents from outside of Oak Bay and the congregation than in this group. 
·         The report notes that the “organized opposition group was in full attendance, provided their leaflet and appealed to all entering the Open House to reject the proposed design.” The brochure did not ask people to reject the proposed design. Apart from providing some contextual information, the brochure primarily focused on providing questions that attendees could (and should) ask about the proposal.

 Opinion Poll

 The rationale provided for conducting a public opinion poll was: “Because attendance (at the open houses) was self-selective, the Open House survey can only be considered representative of attendees' views. The report notes that the open house survey “was not expected … (to) be statistically neutral because; (i) attendees self-selected; and, (ii) attendance was reliant on seeing the advertising, reading the flyer/email or word-of-mouth. Attendance is thus more likely to be biased towards those with strong opinions and will not fully represent those with relative indifference to the project – which is expected to be the majority in Oak Bay.
An Opinion Poll was therefore subsequently commissioned separately to better assess community opinion.” “The goal of this was to determine whether the Open House survey was representative of Oak Bay residents' opinions, since these would be relevant to Oak Bay Council and its decision whether or not to approve the proposal.”
R.A. Malatest & Associates were contracted to undertake the telephone-based (landline/mobile) survey – with 410 responses were completed. A random sample of 3,079 listed Oak Bay phone numbers were called – 410 responses were completed. To be eligible to participate, respondents needed to indicate that they were 18 years or older, and that they lived in Oak Bay (verified by asking for their postal code). The report indicates that “tight timelines for this project required calling during the day on weekdays”  - yet calls were in fact made in the evening.

 Poll Questions

 #1 – How familiar are you with the project?

In terms of familiarity with the project: the majority of respondents (71.2%) stated that they were either "Somewhat Familiar" or "Very Familiar" with the project. The survey did not collect information related to how respondents had learned about the project or what their sources of information were.

Very familiar – 19.5%; somewhat familiar – 51.7%; not very familiar – 15.6%; not at all familiar – 12.9%, not sure/don’t know – 2%.

Note: a considerable portion of respondents (28.5%) are not very familiar or not at all familiar with the project. This will have influenced the results.

 #2 – Do you use any community services provided by faith -based organizations?

Respondents' use of faith-based community services might have influenced responses. Respondents were asked whether they used any community services provided by local faith based organizations such as thrift stores, meeting spaces, seniors or children’s services, grief counselling, AA, etc. Over one-half of respondents (57.1%) did not use any community services. Two-fifths of respondents (41.2%) reported using community services to some degree.

Many of these services – 1.2%; some of these services – 15.6%, very few of these services – 24.4%; none of these services – 57.1%; not sure/ don’t know – 1.7%

Note: This information could be used to counter the church’s claim that it provides $2.5 million worth of services to the community – who is using them?

(Note: if a person only used 1 service this option was not available.)

 #3 – Affordable housing is needed in Oak Bay

This section of the report presents respondents’ opinions on the OBUC affordable housing project.

Two-thirds of respondents (66.6%) "Agreed" or "Strongly Agreed" with the statement "Affordable housing is needed in Oak Bay" i.e. 84.6% were neutral or supportive of the need for affordable housing in Oak Bay. Only 12.5% of respondents indicated disagreement. This suggests that a sizable majority of Oak Bay residents agree that affordable housing is needed in the community.

 #4 – Oak Bay Council should act on the Official Community Plan and ensure affordable housing is built"

Over two-thirds of respondents (68.1%) "Agreed" or "Strongly Agreed" with the statement "Oak Bay Council should act on the Official Community Plan and ensure affordable housing is built", i.e. in total, 80.9% are neutral to or wish Council to act on the OCP. Only 12.4% of respondents "Disagreed" or "Strongly Disagreed". This indicates that a sizable majority of Oak Bay residents agree that Oak Bay Council should act on the Official Community Plan and ensure affordable housing is built.

 #5 – OBUC affordable housing project will help address the need for affordable housing in the area"

Nearly two-thirds of respondents (61.2%) "Agreed" or "Strongly Agreed" with the statement "The Oak Bay United Church affordable housing project will help address the need for affordable housing in the area". One-fifth of respondents were "neutral" or "didn’t know" (19.5%) and one-fifth of respondents (19.3%) "Disagreed" or "Strongly Disagreed". The results indicate that a majority of respondents agree the OBUC project will help address the need for affordable housing, while a smaller proportion of respondents do not agree.

  #6 – Do you support the Oak Bay United Church affordable housing project?

Respondents were asked whether they support the OBUC affordable housing project. 52.0% indicated that they support the project and 23.7% were neutral, i.e. 75.7% of respondents were neutral or supportive of the project as designed. 20.0% indicated that they do not support the project. 4.4% indicated they did not know. This means four times as many are supportive to or neutral to the project than object to it.

Note: It is very interesting that only 52.0% indicated they supported the project. That is hardly overwhelming support! 

NOTE: The wording of this question is not clear enough to get a reliable response: it mentions only that the OBUC project is an ‘affordable housing project’, it does not mention the neighbourhood context or the scale of the development. This is important given the fact that only 19.5% of respondents are very familiar with the project.

Opinion Poll Report Conclusions

For Oak Bay United Church, and Council in considering to approve an application, the main questions relate to whether there is objection to the proposed plans and the extent of objections. Of those that expressed a clear preference (i.e. excluding those that did not know):

·         84.6% were neutral to or agreed that affordable housing is needed in Oak Bay. Over five times as many respondents support or strongly support the need for affordable housing as those that don't think it's needed;

·         80.9% were neutral to or support Council implementing the OCP's mandate to approve affordable housing. Again, over five times as many respondents support or strongly support Council taking action on affordable housing as provided for in the OCP, compared to those that don't think action is needed;

·         71.9% were neutral to or agree that the church's project is aligned with affordable housing delivery. Almost four times as many are ambivalent or agree that it is suitable than consider it not aligned.

·         75.7% were neutral to or support the church's project. Again, almost four times as many are ambivalent or support the project than oppose it.

·         Respondents appear not to be influenced by their use of services and were almost equally split whether they used faith-based community services or didn't.

 Comments on the Surveys

 The public opinion poll was intended to be statistically neutral (where the open house survey was not), to determine how representative of Oak Bay resident' opinions the Open House results were, and to better assess community opinion.

 The problem is that, with the exception of the questions concerning support for affordable housing, the public opinion poll did not cover the same territory as the open house surveys – and, therefore, does not corroborate the open house findings. It was a waste of money. It failed to probe about particular aspects of the OBUC project (e.g. height, massing, number of units, traffic and parking). The contextual information (not provided in the Public Opinion Surveys Report, only made reference to multiple buildings in the area, failing to mention the surrounds on 3 sides of 1-2 storey houses.  For those only somewhat familiar (51.7%), not very familiar (15.6%), not at all familiar (12.9%), or not sure/ don’t know (2%) =  82.2% of respondents, this omission is disturbing and makes the findings questionable. 

information !!!